12 of 13 AI models knew the new name and still wrote the old one

A recent benchmarking study on Kaggle, titled 'Semconv Drift,' tested 17 AI models on their ability to generate accurate OpenTelemetry instrumentation code. The results reveal a significant discrepancy between model knowledge and output: 12 out of 13 models correctly identified updated attribute names when asked directly, yet still utilized retired, deprecated names when generating code. The study highlights that models often prioritize common, outdated patterns found in training data over current specifications. Even when prompted with version-specific instructions, models showed only a 36% reduction in errors, sometimes inventing non-existent names instead. The author concludes that relying on AI for technical tasks requires rigorous validation, as models can produce 'half-fixes'—such as updating units while keeping deprecated names—that pass code reviews but break telemetry pipelines. The findings suggest that developers should implement automated linting against official specifications rather than trusting AI-generated code blindly.