Key takeaways
- Inventory every workflow, prompt, tool contract, schema, policy, cache, evaluation, and customer surface tied to the current model.
- Compare versions on frozen regression cases and fresh production-like cases, including refusals, ambiguity, and tool failures.
- Shadow, canary, and stage the rollout with explicit thresholds and a tested rollback path.
- Publish an internal change record covering intended use, limitations, evaluation, incidents, cost, latency, and ownership.
01
“Drop-in replacement” is a billing label, not a testing result
A new model follows instructions better but uses different JSON edge cases. It is faster on short prompts and slower with tools. Its refusals change, its summaries sound more certain, and a prompt workaround now creates repetition. The API call still succeeds while the operating behavior has moved.
Start with an inventory: workflow, users, decisions, data, prompt and template versions, retrieval, tools, schemas, permissions, guardrails, caches, model settings, expected outputs, evaluations, downstream systems, and owner. Rank migrations by customer exposure and recovery cost.
Swipe to compare every column
| Test layer | Compare | Release blocker example |
|---|---|---|
| Behavior | Accuracy, grounding, refusal, tone | Unsupported claim rate rises |
| Contract | JSON, tools, ordering, idempotency | CRM write shape changes |
| Operations | Latency, cost, rate limits, retries | Tail latency breaks journey |
| Risk | Privacy, injection, permissions, cohorts | Cross-tenant or policy failure |
02
Freeze the important cases, then add new ones
Keep a versioned regression set with common work, high-value cases, ambiguous input, no-answer cases, unsafe requests, prompt injection, stale sources, tool errors, multilingual variants, and known incidents. Preserve expected constraints and acceptable ranges rather than demanding one exact sentence.
Add recent, redacted production cases so the suite does not become a museum of last year’s failures. Review disagreements by impact and pattern. A small average improvement should not overrule a severe regression in a rare but consequential workflow.
03
Observe before the new model receives authority
Run the new version in shadow where policy allows, compare its proposed outputs, and prevent external actions. Then use a narrow canary by internal users, tenant, market, or low-risk workload. Expand only when quality, safety, cost, and latency remain inside agreed thresholds.
Trace model version, prompt, retrieval build, tools, policy, and outcome. NIST’s AI RMF includes post-deployment monitoring, override, recovery, decommissioning, and change management. Decide the rollback trigger and preserve the old path before the rollout starts.
04
Leave a model card for the system you actually operate
The model-cards paper proposes transparent reporting of intended use, performance, evaluation conditions, and limitations. Extend that discipline to the application: why the migration happened, what changed, which cases were tested, results by meaningful group or workload, known gaps, owner, approvals, deployment stages, and incidents.
Review prompt workarounds after migration; some are obsolete and others conceal a regression. Retire old credentials, endpoints, caches, and scheduled jobs only after rollback risk has passed. A migration is complete when the organization can explain the new behavior and recover from it—not when traffic reaches one hundred percent.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
- Mitchell et al.: Model Cards for Model Reporting
- NIST: AI RMF Core
- NIST: TEVV framework for evaluating AI systems
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
