If you’re running LLMs in production, this one dependency has no version management story: the one generating your outputs.
Every software dependency you use gives you a version number. That version number comes with a changelog, a lockfile entry, and a diff you didn’t have to ask for. If a library update breaks something, you know immediately and you roll back. Model providers give you a version string you can pin, but no changelog, no automatic diff, and no notification when behavior changes. When OpenAI ships gpt-5.5 or Anthropic releases Claude Sonnet 5, you find out when something downstream breaks. Or you don’t find out at all.
In a previous Field Notes piece, I built a governance layer that measures whether AI security reviews are consistent enough to trust. One of the design decisions was treating model selection as a governance variable: switching models changes what gets found. I wanted to explore the follow-up question here. What happens when you don’t switch models, but the model switches on you?
What the data showed
Finding 1: The judgment tier drifted
Severity calibration, actionability, and verdict completeness all shifted across version updates. Correctness scores held through each run, but the drift showed up in the things that mattered for decisions.
Severity calibration. GPT-5.4 caps at High across all three review prompts. GPT-5.5 introduced Critical on one of them. Both Sonnet versions use Critical across all three. The calibration shifted partially, which is harder to handle than a clean break. A downstream system routing on severity now behaves differently depending on which version it’s running against, and nothing in your codebase changed.
| Prompt | GPT-5.4 | GPT-5.5 | Sonnet 4.6 | Sonnet 5 |
|---|---|---|---|---|
| tf_security_review | High | High | Critical | Critical |
| devops_code_review | High | High | Critical | Critical |
| devops_diff_review | High | Critical | Critical | Critical |
The architecture tradeoff is a separate finding. Version updates didn’t change either provider’s recommendation: both GPT versions recommended SQS + Lambda throughout, both Sonnet versions recommended Redis + Sidekiq. But the two providers gave opposite answers to the same scenario. Version stability within a provider is not the same as reliability across them.
Finding 2: Before we could measure the drift, the tooling broke
Sonnet 5 broke the capture script. GPT-5.5 drove a token increase dramatic enough to expose any production ceiling you hadn’t accounted for. The failures looked different but the root cause was the same: a version update changed behavior in ways the tooling wasn’t built to handle.
Response shape changed. Sonnet 5 returns a ThinkingBlock as the first element in message.content. The script assumed content[0] was always a TextBlock. Hard crash. Zero output. The fix was three lines:
# Before: assumes text is at index 0
"output": message.content[0].text,
# After: finds the first TextBlock regardless of position
text_block = next(b for b in message.content if hasattr(b, "text"))
"output": text_block.text,| Judgment Prompt | GPT 5.4 → 5.5 | Sonnet 4.6 → 5 |
|---|---|---|
| tf_security_review | Changed | Changed |
| devops_diff_review | Changed | Same |
| devops_architecture_tradeoff | Same | Same |
Methodology
Models: GPT-5.4, GPT-5.5, Claude Sonnet 4.6, Claude Sonnet 5.
Prompt suite: 9 prompts split into two tiers, structural and judgment.
Scoring: Correctness and Production Readiness on a 1-5 scale. Judgment prompts additionally scored on verdict, severity ranking, and Decision Changed.