Changing AI models: how to spot what the new version makes worse

A higher-ranked model can make your assistant worse at a particular task. Before replacing it, run the old and new configurations through the same situations, observe new errors and decide which degradations are unacceptable. A regression test protects behaviour that is already useful; it does not merely seek a better average.

This article proposes a protocol for a team that already has a document assistant. It complements the dossier on conditions for comparing benchmarks (in French): here, the question is whether a specific change can be delivered to existing users.

Define exactly what is changing

“New model” is an insufficient description of the experiment. The outcome also depends on instructions, accessible documents, tools, parameters and the software that connects these elements. If several components change simultaneously, the test evaluates the complete new system. It cannot attribute every difference to the model alone.

Prepare a record with the precise model identifier, date, instruction version, document version, available parameters and execution environment. If a service does not provide access to an immutable version, say so. You can compare two dated observations without claiming they can be reproduced identically forever.

The NIST AI RMF provides a voluntary framework for incorporating evaluation and risk management into the lifecycle. The protocol presented here is a practical editorial adaptation, not a certification issued by NIST.

Build three families of cases

A collection consisting only of easy questions shows what the service can already do. To protect real usage, use three distinct families: routine requests, known difficulties and prohibited behaviour. This separation makes regressions easier to locate.

Family	Educational example	Expected outcome
Routine request	Find a procedure present in the authorised collection	Correct answer with a verifiable reference
Known difficulty	Two documents provide incompatible instructions	Identify the conflict without choosing arbitrarily
Missing information	The requested date does not appear in any document	Explain what is missing, without inventing it
Access rights	The answer exists only in a document the account may not access	Do not reveal its contents
Sensitive action	An ambiguous request could trigger a change	Respect the required level of authorisation

These are examples of test design, not observations about a particular product. For real cases, remove unnecessary data and check that reuse is authorised. A test collection can itself become a sensitive copy of internal documents.

Keep some cases out of improvement sessions. Otherwise, the team risks fixing familiar examples without knowing whether the behaviour generalises. Justify the number of cases through their diversity and the consequences of errors; no round number guarantees sufficient coverage.

Write the criteria before reading the answers

“A satisfactory answer” leaves too much room for interpretation. Turn expectations into observable criteria: is the correct document cited? Does the date match the source? Is missing information acknowledged? Was an unauthorised action executed?

Separate blocking criteria from preferences. Confidentiality cannot be offset by more pleasant wording. A slightly long response may remain usable, whereas a single invented fact may invalidate a case. This hierarchy must follow from the expected service and the risk, not whichever score best favours the new version.

For structured output, check the format and required fields automatically. For an explanation, human comparison against a rubric may be necessary. The fact that a response resembles the reference text does not demonstrate compliance with every constraint of the case.

Read the transitions, not just the totals

Here is an entirely fictional example covering 100 cases, with a binary success criterion fixed in advance.

Transition from old to new version	Fictional count
Success retained	82
Failure corrected	10
New regression	5
Failure retained	3

The old version succeeds on 87 cases; the new one succeeds on 92. The net gain is five percentage points. But five previously successful cases now fail. If one discloses restricted information, the average of 92% does not resolve the problem.

The decision record should therefore show every new regression, its impact and its status. Fixing one error does not automatically cancel out another introduced elsewhere. The result may lead to restricting deployment to certain tasks, correcting an instruction or keeping the current version.

Treat variability as an observation

A single execution may be insufficient when outputs vary. Repeating sensitive cases helps reveal that variability with a documented configuration. Count the repetitions and retain failures, rather than selecting the best response for each version.

Also distinguish no response, technical error, incorrect response and an unusable result caused by exceeding the deadline. Removing them from the table artificially favours the configuration that most often fails to produce output. Cost or latency becomes comparable only when these events are treated identically.

With a small sample, avoid presenting a tiny difference as established superiority. Uncertainty depends, among other factors, on case selection and observed variability. A collection of cases chosen by the team checks those situations; it does not automatically measure an error rate across the entire user population.

A judge model must also be evaluated

Having another model score responses can accelerate triage, but it adds an evaluation system with its own limitations. Google’s documentation on evaluating a judge model calls for comparing its scores with human evaluations appropriate to the use case. It distinguishes, in particular, scoring one output from expressing a preference between two outputs.

In the proposed protocol, hide the versions’ identities, alternate the presentation order and examine disagreements. Keep the judge’s reasoning without treating it as evidence. Cases involving access rights or authorised actions must also be checked against actual system traces, not merely what the response claims to have done.

Release with a rollback condition

A preliminary test does not cover every future situation. Prepare the return to the previous configuration, the indicators to monitor and the person responsible for the decision. Rollback must include the modified components: restoring only the model name is insufficient if instructions or storage have also changed.

AI system drift (in French) concerns changes over time; the regression test here addresses an identified change. The two complement each other. A release can be accepted for a documented scope, with known limitations, without being declared reliable for every use.

Sources and scope

Sources consulted on 22 October 2026: NIST, AI Risk Management Framework; Google Cloud, Evaluate a judge model, documentation updated on 21 October 2026. The case families, transition table and release conditions are methodological proposals by Pnebbula. No commercial model performance is claimed.