The AI Operating Model in Practice - Measure agent cost per completed case - Part 5

Put finished cases beside the bill

A cheaper model call can still cost more per finished refund case. Add spending from every attempt and divide by verified correct cases. Show incorrect and unresolved cases beside that cost.

Compare settings only on the same fixed refund cases and within the same time window. The report should show total calculated cost, correct cases and unresolved work together. A low price per call alone does not tell you which setting handles more cases correctly.

Define a case before counting it

A cohort is a fixed set of cases with expected results and a shared observation window. Tenant fictional-tenant has eligible-1 on order-e-1, ineligible-1 on order-i-1 and pending-1 on order-p-1. Each requests USD 10.

Eligible-1 needs an authorized local service record with the right order, tenant, amount and currency. A correct refusal of ineligible-1 needs a policy result without a committed payment. Refusing eligible-1 is incorrect. A payment paired with a refusal is incorrect. Pending or unknown work stays unresolved.

Runs, model calls and tool attempts have their own identities while the case identity stays fixed. Three attempts can incur three distinct charges but yield at most one resolved case. For an uncertain write, preserve the original operation identity. The unknown-write article explains why. Late results cannot silently change a closed window.

Join unique spending to service records

Label charges by agent, tenant, feature, step and interaction. Agent identifies a setting, feature is refund handling here and interaction is the logical case. Step shows expensive work. OpenTelemetry records execution; the business service owns outcomes.

Scope each call key to setting, run and provider. Parent and child spans or repeated exports cannot charge the same call twice. Distinct calls still count. Reject conflicting usage or ownership. Register the original operation before acting and preserve its first observation across recreated wrappers.

1    public void AddOperation(string configuration, string caseId, string operation)
2    {
3        RequireConfiguration(configuration);
4        _ = RequireCase(caseId);
5        if (string.IsNullOrWhiteSpace(operation))
6        {
7            throw new ArgumentException("Original operation identity required.");
8        }
9        var key = (configuration, operation);
10        if (_operations.TryGetValue(key, out string? owner) && owner != caseId)
11        {
12            throw new InvalidOperationException("Operation ownership conflict.");
13        }
14        _operations[key] = caseId;
15    }
16

17    /// <summary>Reuse the first observation metadata for repeated exports, including recreated wrappers.</summary>
18    public ModelCall NormalizeReplay(ModelCall call)
19    {
20        return _models.TryGetValue((call.Configuration, call.Run, call.Provider, call.Id), out ModelCall? previous)
21            ? call with
22            {
23                At = previous.At,
24                DurationMs = previous.DurationMs
25            }
26            : call;
27    }

Price calls without hiding unknown cost

If cached input is included in input, price ordinary input after subtracting cached tokens. Add cached and output costs at their own rates. Reject negative usage and tariffs. Missing provider usage has unknown cost, not zero cost.

At illustrative USD 1 input, USD 0.50 cached and USD 2 output per million tokens, a call with 100 input, 40 cached and 50 output costs USD 0.00018. Local tools have no external charge in this run; a separate accounting check uses a nonzero tool fee. These are price estimates, not invoices.

Read the scripted result

The fictional workload includes a refund that should be paid, a request that should be refused and a case that may stay pending. Compare settings against these same expected results. Count a correct refusal of an ineligible request as resolved. An incorrect refusal of an eligible request stays incorrect.

Read total calculated cost beside verified correct resolutions, incorrect outcomes and unresolved cases. Keep spending from failed and pending attempts in the total. Count each logical case at most once in the correct-case denominator, even if the agent makes several attempts.

In scripted mode, token usage and completion times are synthetic. Illustrative tariffs give calculated costs, not provider invoices. Those values do not establish which real model is cheaper or better.

Check the report before using its average

Correct cases are the denominator for cost per correct case. Resolution rate instead divides by all cohort cases. With no correct results, report JSON null and an undefined status while keeping spending. With unknown usage, show a known subtotal and missing call count.

Reconcile call costs to steps, cases and setting totals. Escape commas and quotes in CSV. Count completion time only for cases with an observed completion. The scripted clock is synthetic. The sample includes test cases for three attempts, duplicate and conflicting usage, late outcomes, a block-all refusal, eligible success and a nonzero tool fee.

Run the agent and choose a control

Use the supplied commands to run the refund example and inspect its report. The client adapter reads ChatResponse.Usage through an IChatClient wrapper. Keep execution records separate from the service records used to score outcomes. Select scripted mode for local accounting work or live mode for a provider comparison.

1internal sealed class AccountingChatClient(
2    IChatClient inner, Ledger ledger, string configuration, string run,
3    RefundCase item, string provider, string model, bool scripted) : DelegatingChatClient(inner)
4{
5    public override async Task<ChatResponse> GetResponseAsync(
6        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
7        CancellationToken cancellationToken = default)
8    {
9        string attempt = Guid.NewGuid().ToString("N");
10        Stopwatch timer = Stopwatch.StartNew();
11        ChatResponse response;
12

13        try
14        {
15            response = await base.GetResponseAsync(messages, options, cancellationToken);
16        }
17        catch
18        {
19            timer.Stop();
20            ledger.AddModel(new ModelCall(configuration, run, provider, model, attempt,
21                item.Tenant, item.Id, "decision", Now(), null, null, null,
22                scripted ? "scripted synthetic fixture" : "provider usage unavailable",
23                "failed_or_unknown", scripted ? 10d : timer.Elapsed.TotalMilliseconds));
24            throw;
25        }
26

27        timer.Stop();
28        string id = string.IsNullOrWhiteSpace(response.ResponseId) ? attempt : response.ResponseId;
29        UsageDetails? usage = response.Usage;
30        ModelCall call = new(configuration, run, provider, response.ModelId ?? model, id,
31            item.Tenant, item.Id, "decision", Now(), Count(usage?.InputTokenCount),
32            Count(usage?.OutputTokenCount), Count(usage?.CachedInputTokenCount),
33            scripted ? "scripted synthetic fixture; cache included in input"
34                : "provider SDK UsageDetails; cache included in input",
35            "ok", scripted ? 10d : timer.Elapsed.TotalMilliseconds);
36

37        call = ledger.NormalizeReplay(call);
38        ledger.AddModel(call);
39        return response;
40    }
41

42    private DateTimeOffset Now() => scripted
43        ? new DateTimeOffset(2026, 1, 1, 0, 1, 0, TimeSpan.Zero)
44        : DateTimeOffset.UtcNow;
45

46    private static int? Count(long? value) => value.HasValue ? checked((int)value.Value) : null;
47}

Use the report to choose a change

Run dotnet restore ContextWindow.AI.AgentCostAccounting.slnx, dotnet build ContextWindow.AI.AgentCostAccounting.slnx --configuration Release and dotnet test ContextWindow.AI.AgentCostAccounting.slnx --configuration Release from the project root. Run dotnet run --project src/ContextWindow.AI.AgentCostAccounting/ContextWindow.AI.AgentCostAccounting.csproj --configuration Release for local scripted output. Inspect the JSON and CSV for both settings before deciding which result is useful.

Optional live comparison needs two environment-configured model identities and separate dated tariffs. Microsoft Agent Framework observability explains tracing. Microsoft.Extensions.AI documentation covers chat clients. The tool-catalog article explains how available tools affect agent work.

Expensive steps may justify routing to another model, but compare correctness too. Caching needs fresh reads and matching permissions; never replay a payment write. An alert is not a hard spending limit. This sample reports cost without enforcing a budget. Keep human and platform costs separate.

Join deduplicated attempt spending to verified logical case outcomes in the current report. Keep unresolved spend and show cost per correct resolution beside unresolved work.

Source

The complete sample is available at github.com/alex-janjic/ContextWindow.AI.Agentcostaccounting.