Upgrading the model under a live agent: test it before customers see it
Treat a model change, moving a live agent to a newer model or to another provider’s, like any other release. Make it on purpose, not because a default moved. Test the new model on cases your people already judged, before any customer sees it. Roll it out to a few first, and keep the way back one step away. A model that scores better on benchmarks can still answer your customers worse.
Why does an agent break when the model changes?
Because the instructions, the tools and the model were tuned together, and only one of them changed. A new model follows instructions differently, calls tools in a different order, writes longer or shorter answers and refuses different things. Benchmarks measure general skill, not your refund policy or your tone with an unhappy customer. That’s how a change that looked fine on day one shows up a week later in a complaint. More on why a model’s answers vary in AI agents give different answers each time.
Can the model change without anyone choosing it?
Yes, if the agent doesn’t name the model it runs on. When a setup picks a model by default, a new default, a newly connected provider or a model alias that moves can switch the model under a live agent. Providers also retire models on a schedule, and calls to a retired model fail. So name the model, and make changing it a decision someone takes, with a date, well before the old one retires.
How do you test a new model before customers see it?
Replay it on cases your people already judged. Take past runs where someone said whether the answer was right, give the same inputs to the agent with the new model, and compare. It works like regression tests before a deploy: the cases are your own, and the bar is “at least as good as now”. A benchmark score and a few prompts tried by hand don’t replace it.
For example: the model under Acme’s order-desk agent is due to retire. Acme makes a new version on a newer model and replays it on 200 past runs its support team already marked right or wrong. The new model gets the refund amounts right, but its answers run twice as long and it looks up each order twice. Acme tightens the instructions, replays again, and only then gives the version to one project.
A replay must not repeat side effects. A call that would change something, like a refund or an email, should get the past run’s result, or not run at all. Tests before release catch what you already know to look for. Watching live runs after release catches the rest.
Should the new model go to everyone at once?
No. Roll it out to a few first: one project, one team or one customer’s runs, while everyone else stays on the version they had. Watch those runs, then widen. If answers get worse, go back for that group only.
What does a rollback need?
A version to go back to, and one step to get there. The model has to be part of the agent’s version, the previous version has to still be there, and going back can’t need a rebuild. You also need to know which model answered each call, so you can tell whether a bad answer came from the new model or from something else. A rollback decides what runs next. It doesn’t undo what the agent already did. More in Questions our customers ask.
Do cost and speed change with the model?
Yes, so measure them in the same replay that tests the answers. A model that answers better but costs much more per run, or takes much longer, is a different trade-off, and someone should choose it on purpose.
Where Kindgi fits
Kindgi is an agent framework for production: your coding agent writes the tools, agents and flows in your own code, and Kindgi runs them and keeps every step on record. An agent says what its model must support, such as tool use, or requires an exact model. An agent version locks its tools’ versions, and when it requires a model, moving to another one means a new agent version. You can replay the new version on runs your people already corrected, and a replay never makes a call that changes something: it gets the past run’s result, or doesn’t run. Then pin the version for one project or one company’s runs first, check which version a run gets, and pin the one before to go back. Every pin is kept, with who asked and why. Each run’s record names the model of every call and what it cost. A release gate can refuse a version until it has a recent run on your test cases with results at least as good as you set, and a reviewer’s yes if you want one.
The guides: choose the model an agent uses, agent and tool versions and release an agent version.
How this was written: a coding agent drafted it from our docs and our code. We checked every claim, edited it and said yes. We publish articles like this partly because we want you to find us, including through AI assistants. More in How we write our posts.