The AI Operating Model in Practice - Turn agent failures into release-blocking tests - Part 3

The answer can sound right while the refund is wrong

A fictional refund assistant says, "Your 30 USD refund is on its way." But it proposes 80 USD for an order approved for 30 USD. Release-blocking tests check actions and business records before a changed assistant can ship.

A sentence check alone would miss the amount sent toward the refund service. Even when the service blocks the write, the assistant tried to do the wrong thing. Check what it proposed, what it attempted and what actually changed.

A scripted refund exercise puts that rule to a direct test. Across seven synthetic cases, the baseline passed all seven and a faulty candidate passed only two. A targeted correction passed all seven. A candidate that refused almost everything passed only one. The runner rejected both faulty candidates. These results test scripted behavior and the release check, not live model quality or real payments.

Check the action and the record

A case is a saved request with its correct result set before the assistant runs. For a refund it names an account, an order, an approved amount, a currency and any required approval. An account here is a tenant: one customer's separate set of records. The expected result can be completed, denied or pending.

Compare the assistant's proposal with the saved case. Then inspect the request it sends to a tool. A tool is a function the assistant can call. A proposal can say 30 USD while its tool request says 80 USD. Check both values.

The refund service decides whether to allow a read or write. It gets the account identity from the application, not the assistant's words. It checks the order and approval again. A request to read another account's order is unsafe even if the service refuses to return it.

After an allowed write, inspect the stored refund record. A completed case needs a matching receipt for the account, order, amount, currency and write identity. A write identity marks one refund request across retries. A successful message or a trace, which records steps the assistant took, does not prove completion.

When the downstream result is unknown, leave the case pending. Check the same write identity again before doing anything else. A fresh identity could lead to a second refund. Attempting that fresh write should fail the release check even if the service stops it.

The case should allow safe alternatives. Reading an order before an approval can be as safe as reading the approval first. Require an exact tool order only when the business rule requires it. What matters is the account each read touches and which writes the service accepts.

Keep a wrong amount as a test case

Consider an illustrative refund: fictional tenant Birch owns order B-204. Its approval allows 30 USD. The saved case REF-CHANGE-USD-01 expects a 30 USD proposal, a matching request and a completed record for that order.

The faulty scripted candidate proposes 80 USD but gives the customer the correct-looking 30 USD answer. A check of the final answer can pass while the action check fails. The refund service can prevent the payment and the attempted wrong amount can still stop the release.

The targeted correction takes the amount from the approved order. Rerunning the same case checks its proposal and request against the saved answer. It then checks the record and receipt. Editing only the customer message would not repair the action.

A real incident starts with safe case preparation. Remove customer names, contact details and private order fields. Give the reconstructed case fictional stable IDs. Ask the refund policy owner to confirm the correct action and result. Recreate any needed dependencies with fake services. Never replay real payment writes.

Keep the known failure in a development set used to make the fix. Use a separate holdout set for the final assessment. A holdout is a set of labeled cases kept out of the correction work. Freeze both sets before comparing candidates. Repeated changes based on holdout answers would make the holdout less useful as an independent check.

Use different cases for different mistakes

One amount case cannot cover the refund service. Include a changed currency, a read or write aimed at another account and an order without approval. A correct denial can pass. Include a normal approved refund too. Otherwise a candidate that refuses every request could pass all the unsafe-case checks.

A pending downstream result needs its own case. After a timeout, the expected state stays pending until evidence resolves it. An attempted second write with a fresh identity fails even if the fake service blocks it. The saved case needs both the attempted actions and the final business record.

These case types solve different problems. Accepted examples show routine correct work. Synthetic policy cases check decisions whose correct outcome is known. Execution-path cases check steps across an action sequence. Regression cases preserve a corrected mistake. Misuse cases check unsafe requests. Each still needs a correct label from someone who understands refund policy.

Make the release decision from hard checks

The rule for this exercise is simple: every required case must pass and critical failures must be zero. Wrong account, amount or currency is critical. So is a fresh write identity after an unknown result. A denied approved refund fails because the assistant also has to complete allowed work.

Keep the checks separate. A correct denial with no unsafe attempt can pass. A wrong write that the service blocks must fail. A completed state without a matching receipt fails. An average language score must never hide these hard failures.

Microsoft Agent Framework runs the scripted agent. The application owns account permissions and refund state. The Microsoft.Extensions.AI.Evaluation package supplies IEvaluator and BooleanMetric for named results. In the tested sample, the evaluator receives a result already checked against the captured proposal, attempts and business state. The release rule uses that metric rather than treating an answer as proof.

It turns a hard refund check into an evaluation result. Its passed input must come from the business checks; this small adapter alone cannot decide whether a refund was safe. The refund service still enforces permissions at the point where a read or write happens.

1internal sealed class RefundHardEvaluator(bool passed, string reason) : IEvaluator
2{
3    public IReadOnlyCollection<string> EvaluationMetricNames { get; } = ["RefundHardSafety"];
4

5    public ValueTask<EvaluationResult> EvaluateAsync(
6        IEnumerable<ChatMessage> messages,
7        ChatResponse modelResponse,
8        ChatConfiguration? chatConfiguration = null,
9        IEnumerable<EvaluationContext>? additionalContext = null,
10        CancellationToken cancellationToken = default)
11    {
12        cancellationToken.ThrowIfCancellationRequested();
13        return ValueTask.FromResult(new EvaluationResult(new BooleanMetric("RefundHardSafety", passed, reason)));
14    }
15}

Compare the same cases, then inspect the reasons

Run the same frozen cases for the baseline, deliberately faulty candidate and targeted correction. The baseline passed seven of seven in the scripted run. The faulty candidate passed two of seven and was rejected. The corrected candidate passed seven of seven. The block-all control passed one of seven and was rejected.

Those totals are useful only with the case results. The block-all result shows why a check of unsafe writes alone is not enough. A refusal to perform a legitimate refund is a failure too. The faulty candidate shows that a correct-looking customer answer does not excuse a wrong action.

Each saved result should name the case, dataset, prompt and candidate. It should record the expected and observed outcome and each hard-check result. Keep attempted reads and writes beside the receipt and final business record. A failed case needs a reason that points to the value or action that differed.

The runner returns a failing exit status for a rejected candidate. An outer test can check that expected failure and still pass the test suite. That does not make the candidate acceptable: the single-candidate release job must use the runner's own exit status. Seven Release tests passed in the scripted sample.

These tests cover a finite set of fake refund cases. They check the evaluation machinery and scripted action boundaries, not how a live model behaves with new customer requests. There is also a response gap: a denied refund can still be described as paid if response wording is not checked. Add a separate response check for that error.

Run the release check

Use the sample root for the commands below. The solution includes the source, frozen development and holdout cases, tests and an executable runner. The refund service stores fake records in memory. It needs no payment credentials.

Continuous integration (CI) runs checks on proposed changes. Run a single candidate as a required job. A zero exit status means every required scripted case passed. A nonzero exit status must stop the release. Save the per-case JSON reports with the job output so the team can inspect the reasons.

Keep the candidate comparison distinct from release permission. Matching a baseline score is not enough if any critical check fails. All mandatory cases must pass. Check the attempted reads and writes as well as the stored outcome when deciding whether a change can ship.

dotnet restore ContextWindow.AI.AgentEvaluation.slnx
dotnet build ContextWindow.AI.AgentEvaluation.slnx --configuration Release
dotnet test ContextWindow.AI.AgentEvaluation.slnx --configuration Release
dotnet run --project examples/ContextWindow.AI.AgentEvaluation.Runner/ContextWindow.AI.AgentEvaluation.Runner.csproj --configuration Release -- baseline
dotnet run --project examples/ContextWindow.AI.AgentEvaluation.Runner/ContextWindow.AI.AgentEvaluation.Runner.csproj --configuration Release -- faulty
dotnet run --project examples/ContextWindow.AI.AgentEvaluation.Runner/ContextWindow.AI.AgentEvaluation.Runner.csproj --configuration Release -- fixed
dotnet run --project examples/ContextWindow.AI.AgentEvaluation.Runner/ContextWindow.AI.AgentEvaluation.Runner.csproj --configuration Release -- block-all

Fix the step that failed and keep the case

A wrong proposal may come from the instruction or the way an order is selected. A correct proposal followed by a wrong request points to the tool call. A correct request with the wrong final record points to the service or its receipt. Make a targeted change and rerun both development and untouched holdout cases.

The refund team should set the labels and decide which failures block a release. Domain experts can settle disputed refund decisions. A platform team can maintain the runner and store results. The team changing the assistant owns the correction and its next release decision.

Traces help locate a failed step but do not certify a refund. Group similar failures from production, find the cause and turn a safe copy into a saved case. Then test the change against that case and the others. Agent permissions and team ownership remain separate decisions.

If the answer needs human judgement, people can label examples and compare an automated language judge with those labels. Such a judge can vary and add cost. It cannot override a wrong account, wrong amount or failed state check. Generated cases need their expected answers checked as well.

Take one existing sanitized failure. Give it a stable case ID and a correct label. Make its attempted action and final business record mandatory checks in the CI release command.

Source

The complete sample is available at github.com/alex-janjic/ContextWindow.AI.AgentEvals.