Skip to content
Context Windowby Alex Janjic
Menu
contextwindow.us/posts/the-ai-operating-model-in-practice-budget-for-operating-agents-before-scaling-them-part-6Article

The AI Operating Model in Practice - Budget for operating agents before scaling them - Part 6

A small model bill can hide review, correction and investigation work. A fictional refund example shows how to price those costs, check daily specialist slots and decide when to hold expansion.

AI systems
Source links stay with the relevant section
A small pile of model-call tokens sits beside more numerous review cards, a calendar and a ledger. A path forks at the service decision.

The model bill is only the start

A fictional refund assistant takes 100 requests in a working week. In one planned setup, its model calls cost $45. Review, correction and investigation help bring its full operating cost to $1,145. Every figure here is an illustrative assumption, not a bill or an observed result.

An operating budget pays for running and improving the service. It covers attempts that fail, cases still waiting and the people who check the work. Before adding more requests, ask whether the team can pay for correct outcomes and staff the required checks on time.

Part 5, Measure agent cost per completed case, deals with the result of each case. This article adds service hours and the daily limits on qualified review. A cheap call does not make an unreviewed refund safe.

What goes into the decision

This whole example is fictional. Compare setup A and setup B against the same 100 new cases that enter on Monday 6 April 2026. The closed-start cohort is this fixed group of cases. Both setups use the same eligibility records, expected outcomes, case-level deadline map and acceptance rules. The outcome snapshot is Friday 10 April at 17:00. Money is illustrative US dollars for those five working days.

Eighty requests are eligible and 20 are ineligible. A correct eligible case needs the right refund verified in the business record. A correct refusal of an ineligible case also counts. A wrong refusal of eligible work or an unauthorized refund of ineligible work is incorrect. An unknown write or a pending review is unresolved. Count each logical case once, even after several attempts.

Maya, the product owner, owns the outcome and spending decision. Leo owns engineering execution, releases and the stop control. Noor owns operations coverage and manual service. Imani checks assumed rates and allocation. Sana is the sole qualified specialist for this review schedule. These are fictional names and responsibilities, not an added department.

A proposed refund needs service authorization and any mandatory specialist approval before action. A sent request is an attempt, not a verified refund. Refund principal is money sent to a customer and stays outside this operating budget. The review capacity article explains the wider admission rule.

From request to operating decision

Case intakeStable case and action IDs
Service checksEligibility and approval before action
Business dispatchAttempt the authorized action
Outcome and cost ledgersVerified result or unresolved status
Owner decisionCost, quality and timed review slots
  1. Case intakeproposeService checks
  2. Service checksauthorizedBusiness dispatch
  3. Business dispatchverify or mark unknownOutcome and cost ledgers
  4. Service checkspending reviewOutcome and cost ledgers
  5. Outcome and cost ledgerscost and outcomeOwner decision
Read this diagram as text

Intake assigns a stable case ID and action ID. Service checks test eligibility and obtain mandatory qualified approval before business dispatch. Dispatch attempts the authorized action. The business record verifies its result or the ledger keeps the attempted write unknown. A review still pending at service checks also enters the ledger as unresolved without dispatch. The owner compares ledger spending and outcomes with timed qualified review slots before deciding whether to expand, hold or retire.

Business outcomes, spending and the available review schedule inform the decision.

Count the right result, not just a finished reply

The outcome rows are assumed examples, not service measurements. Both setups begin with the same 80 eligible and 20 ineligible requests. A has 60 correct refunds and 12 correct refusals. It wrongly refuses six eligible requests and refunds two ineligible requests without authorization. Its remaining 14 eligible and six ineligible cases are pending or unknown.

B has 70 correct refunds and 12 correct refusals. It wrongly refuses two eligible requests and refunds two ineligible requests without authorization. Eight eligible and six ineligible cases remain pending or unknown. No incorrect terminal answer counts as a useful correct resolution.

Correct resolution rate divides correct cases by all 100 requests. Incorrect rate divides terminal incorrect cases by 100. Completion rate divides all terminal cases, including incorrect ones, by 100. At Friday's snapshot the open cases stay in every rate's denominator.

Illustrative outcomes for the same intake, eligibility records and Friday 17:00 snapshot. These are assumed results for comparing plans, not delivered service outcomes.
MeasureOutcomeA, illustrativeB, illustrative
Correct eligible refunds / correct ineligible refusalsCorrect eligible refunds / correct ineligible refusals60 / 1270 / 12
Wrong eligible refusals / unauthorized ineligible refundsWrong eligible refusals / unauthorized ineligible refunds6 / 22 / 2
Pending or unknown: eligible / ineligiblePending or unknown: eligible / ineligible14 / 6 = 208 / 6 = 14
Correct / all intakeCorrect / all intake72/100 = 72%82/100 = 82%
Incorrect / all intakeIncorrect / all intake8/100 = 8%4/100 = 4%
Terminal completion / all intakeTerminal completion / all intake80/100 = 80%86/100 = 86%
Flagged cases requiring approval before actionFlagged cases requiring approval before action1216

Price the whole week

The example's numerator covers all attempts by these 100 cases and the recurring workflow work assigned only to them during this week. That includes incorrect, failed and pending cases. Its denominator is verified correct resolutions in the same cohort and window. If there are none, the ratio is undefined although costs remain.

Cash figures below are forecasts of added service charges, not actual invoices. Staff figures price time from people already paid. They are economic allocations, not extra weekly cash or claimed savings. Free capacity is time available for a plan, not a charge. All prices, hours and rates are fictional assumptions.

Model charges include retries. The $30 hosting share uses reserved capacity; $10 of external tool charges uses request volume. The $10 monitoring subscription is a different service. Imani checks these allocations during the counted operations monitoring time with Noor. Maya checks the budget during the counted support hour. Neither check adds a second hour or charge.

At assumed loaded rates, Sana's review costs $60 an hour. Operations work costs $50 an hour and engineering work costs $80 an hour. Each assigned hour belongs to one line only. Review cannot also be manual correction. Leo's evaluation, incident and shared-control hours are separate. Initial rollout is outside these recurring rows.

Fictional forecast for 6 to 10 April 2026 in US dollars. Cash is incremental service spend; staff is allocated existing paid time. Lines do not overlap.
MeasureRecurring line and allocationAB
Model calls and retries; assumed usage cashModel calls and retries; assumed usage cash$20$45
Hosting; reserved service capacity cashHosting; reserved service capacity cash$30$30
External tools; request use cashExternal tools; request use cash$10$10
Monitoring subscription; workflow share cashMonitoring subscription; workflow share cash$10$10
Sana's qualified review; $60/hourSana's qualified review; $60/hour6 hours = $3608 hours = $480
Manual correction; operations $50/hourManual correction; operations $50/hour3 hours = $1502 hours = $100
Operations monitoring; $50/hourOperations monitoring; $50/hour2 hours = $1002 hours = $100
Evaluation upkeep; engineering $80/hourEvaluation upkeep; engineering $80/hour2 hours = $1602 hours = $160
Incident investigation; engineering $80/hourIncident investigation; engineering $80/hour2 hours = $1601 hour = $80
Shared-control upkeep; engineering $80/hourShared-control upkeep; engineering $80/hour1 hour = $801 hour = $80
Training and service support; operations $50/hourTraining and service support; operations $50/hour1 hour = $501 hour = $50
Cash + assigned staff = recurring totalCash + assigned staff = recurring total$70 + $1,060 = $1,130$95 + $1,050 = $1,145
Recurring total / verified correct casesRecurring total / verified correct cases$1,130 / 72 = $15.69$1,145 / 82 = $13.96
Copyable worksheet. Make one copy per option. Write a source or mark a planning assumption for every input.
MeasureField to copyWhat to enter
Window and case setWindow and case setCurrency; calendar and working hours; stable case and action IDs; intake; eligibility and frozen case deadlines.
Outcome and useful unitOutcome and useful unitCorrect refund; correct refusal; wrong refusal; unauthorized refund; pending review; unknown write. Record correct/intake, incorrect/intake and terminal/intake.
One-time build and rolloutOne-time build and rolloutSource; cash versus exclusive staff hours and rates; separate total; state any later spread across periods.
Cash linesCash linesModel attempts; hosting; external tools; monitoring subscription. Record assumed charge source and one allocation basis for each.
Existing staff linesExisting staff linesQualified review; manual correction; operations monitoring; evaluation; incident; shared controls; training and support. Record distinct role, hours, assumed rate and cost.
Capacity and deadlinesCapacity and deadlinesNamed qualified reviewer; daily hours and prior backlog; fixed case deadline; minutes per case; free slots and admission limit. Separate other staff capacity.
Carry-forward and change testCarry-forward and change testCase/action IDs; old period spend; new period spend; unresolved count; one changed assumption with other inputs fixed.
Owner decisionOwner decisionCash and staff totals; cost/correct case or undefined; owner; stop authority; fundable action and dated reassessment.
Filled fictional planning record. Each alternative runs alone. Time totals and investment sit outside the recurring cost table where indicated.
MeasureRecorded assumptionAB
Case set, working window, currencyCase set, working window, currency100 new IDs; 6 to 10 April 2026; US dollarsSame 100 IDs, period and currency
Eligibility and Friday 17:00 outcomesEligibility and Friday 17:00 outcomes80 eligible, 20 ineligible; 72 correct, 8 incorrect, 20 openSame eligibility; 82 correct, 4 incorrect, 14 open
One-time staff build and rollout, separateOne-time staff build and rollout, separate30 exclusive engineering hours x $80 = $2,40037.5 exclusive engineering hours x $80 = $3,000
Recurring cash forecastRecurring cash forecast$70, assumed service charges$95, assumed service charges
Recurring staff by nonoverlapping roleRecurring staff by nonoverlapping roleSpecialist 6h $360; operations 6h $300; engineering 5h $400Specialist 8h $480; operations 5h $250; engineering 4h $320
New-case specialist time after old backlogNew-case specialist time after old backlog9h free; 6h planned review9h free; 8h planned review
Other roles after other obligationsOther roles after other obligationsOperations 8h; engineering 6h freeOperations 8h; engineering 6h free
Assumed sensitivity, only review minutes changeAssumed sensitivity, only review minutes change40 minutes; 8h; $1,250 / 72 = $17.36 price-only40 minutes; 10h 40m; $1,305 / 82 = $15.91 price-only
Owner, staffing, stop and cost checkOwner, staffing, stop and cost checkMaya; Noor and Sana; Leo; ImaniSame named owners

The daily review slots decide if B can grow

All flagged cases are available Monday. Each case has one fixed review due day in the same deadline map under either setup. The flagged subsets differ, so their daily counts differ. The Friday 17:00 snapshot ends the outcome window; a flagged case can need approval much earlier. Pending accepted work keeps its original due time.

Sana has one review window each weekday from 15:00 to 17:00. On Tuesday, an older cohort's backlog takes 15:00 to 16:00. New cases get Tuesday 16:00 to 17:00. Their other daily windows each hold 120 minutes. That is 540 new-case minutes for the week. Sana is not booked twice.

At 30 minutes per review, A's daily groups are 2, 1, 3, 3 and 3. They run Monday 15:00 to 16:00, Tuesday 16:00 to 16:30 and Wednesday through Friday 15:00 to 16:30. B's groups are 4, 2, 4, 4 and 2. They run Monday 15:00 to 17:00, Tuesday 16:00 to 17:00, Wednesday and Thursday 15:00 to 17:00 and Friday 15:00 to 16:00. Both fit before each group's due time.

Now change only new-cohort review time to 40 minutes. Leave old backlog, due dates, intake, prices per hour and all other assumptions fixed. A needs 480 minutes: 80, 40, 120, 120 and 120 by day. It fits. B needs 640 minutes, or 10 hours 40 minutes. Its daily groups need 160, 80, 160, 160 and 80 minutes. The week has only 540 free minutes, a 100-minute deficit. Monday, Tuesday, Wednesday and Thursday each exceed their own window.

Under these fixed-day assignments, B can admit at most 3, 1, 3, 3 and 2 flagged cases by day. That is 12 of 16. Hold four before action. This is a conservative feasible schedule, not a claim about rearranging deadlines or adding shifts. An unknown route must reserve a possible specialist slot. Earlier review or extra shifts need a new explicit plan.

At $60 per hour, A's 480 review minutes cost $480. B's full 640-minute demand would cost $640 if staffed. The resulting forecast totals are $1,250 and $1,305. Against unchanged hypothetical correct counts, the price-only ratios are $17.36 and $15.91. B cannot deliver that assumed 82-case outcome on this overbooked plan. Held cases remain unresolved, not correctly refunded.

Illustrative one-change sensitivity. Daily demand uses Monday to Friday fixed due groups. B's full-demand price is counterfactual until more qualified slots are funded.
MeasureReview planAB
30-minute baseline daily groups and total30-minute baseline daily groups and total2, 1, 3, 3, 3 = 360m; $3604, 2, 4, 4, 2 = 480m; $480
40-minute daily demand40-minute daily demand80, 40, 120, 120, 120m = 480m160, 80, 160, 160, 80m = 640m
Daily free minutes after older Tuesday backlogDaily free minutes after older Tuesday backlog120, 60, 120, 120, 120 = 540mSame 540m; 100m short overall
40-minute review cost and recurring total40-minute review cost and recurring total$480; $1,250$640 full-demand forecast; $1,305 full-demand forecast
40-minute admitted flagged cases by assigned due day40-minute admitted flagged cases by assigned due day2, 1, 3, 3, 3 = 12; fit3, 1, 3, 3, 2 = 12; hold 4

Keep older work in the right ledger

A stable case ID and action ID connect the proposed refund, approval, attempts and business result. A timed-out dispatch can leave the write unknown. The business record decides whether it happened. The unknown-outcome article covers that handling in more detail.

The Tuesday backlog uses Sana's time but belongs to an older cohort. Its case charges stay in that cohort's ledger. Do not charge those cases again to the new 100. A whole-service period report must still show older-cohort expenses in their own line. The new-cohort comparison has a narrower boundary, stated here openly.

If a Friday case needs work next Monday, keep Friday's spend in Friday's accounts and its cumulative case ledger. Add only new Monday spending to Monday's accounts. Update that same case's outcome when the business record settles it. Do not count a resolved case twice. A weekly ratio for this closed-start cohort is not automatically its lifetime ratio.

What the team can operate

The team can assign costs to the workflow and inspect monitoring records. That does not establish that reviewers agree on hard cases or that the incident rota is ready for more requests. Noor's review coverage, Leo's release checks and a practiced stop action need their own evidence.

Leo can use B's counted two engineering evaluation hours for recurring case checks. They do not prove Sana agrees with another qualified reviewer. Before expansion, Noor must plan a separate nonoverlapping qualified calibration session. A bounded fictional planning allowance is one specialist hour at $60, outside this week's $1,145. It needs approval in a revised budget and a real available slot.

B already assigns one engineering hour to incident work and one to shared-control upkeep. Leo can use those hours to rehearse the stop path and check export of case records for a possible provider change. Export practice does not make switching free. The counted operations support hour covers routine staff guidance, not initial rollout training. Leo should record recurring failures and time to a verified fix as future service measures. Automated diagnosis may suggest a fix but cannot judge its own evidence or authorize release.

Stages describe observable work, not agent counts or tool purchases. Named hours inside B's current total are not added again.
MeasureStageWhat can be shownFictional team, gap and assigned work
ExploringExploringVisible attempts and a small set of checked cases.Leo preserves records using the counted shared-control hour; cost attribution is stronger than this stage.
BuildingBuildingNamed owner, service checks, basic monitoring and qualified review.Maya owns outcomes. Noor checks Sana's dated nine free hours; case monitoring uses the counted operations line.
OperatingOperatingRepeated evaluations, checked releases, corrected failures and incident response.Leo has two counted evaluation hours and one incident hour. Noor must budget one separate $60 specialist calibration hour before claiming reviewer agreement.
ScalingScalingReusable controls and qualified coverage across a larger intake.Not established. Noor must fund timed slots for the 40-minute case and Leo must exercise stop and record export during the counted shared-control work.

Record the choice and the next check

Maya chooses to hold wider B intake in this fictional decision. Its baseline $13.96 per assumed correct case beats A's $15.69 and its assumed outcomes are better. But at 40 minutes the flagged work misses daily slots. Reviewer agreement and the stop path still need checking.

For now, Maya admits only cases that Noor can match to qualified slots before their fixed deadline. Leo stops new actions when those slots are gone and keeps already accepted cases open for service. Noor must count manual handling against real availability. The correction line cannot pay for a complete manual replacement.

One-time build and rollout is another choice. A assumes 30 exclusive engineering hours at $80, or $2,400. B assumes 37.5 exclusive engineering hours at $80, or $3,000. These economic staff investment totals include setup and initial training. They do not include later recurring support hours. This week's unit costs do not spread the investments over weeks. Any later spread needs a stated service life and a separate visible line.

For retirement, Noor needs an agreed manual workload estimate, staffed schedule and funded service route before the change. Leo must reconcile every open case and action ID with the business record. Maya then ends new agent admissions. Stopping software without this service plan would move work into an unfunded queue.

Illustrative choice record. The 40-minute plan is a planning stress check, not an observed quality result.
MeasureOption and accountable ownerEvidence to check and uncertaintyStop authority and work to fundCondition for reassessment
Expand: Maya, product ownerExpand: Maya, product ownerMatched case outcomes and deadlines; review time, agreement and incident readiness remain uncertain.Maya limits admissions. Leo can stop actions. Noor funds extra qualified slots; Leo uses counted evaluation, incident and release time.Approve a bounded increase only when every 40-minute case fits its due day, qualified decisions agree and Leo shows an exercised stop and checked release.
Hold wider B intake: Maya. Chosen.Hold wider B intake: Maya. Chosen.Illustrative B total $1,145 at baseline; 640 review minutes versus 540 free at 40 minutes. Actual outcomes remain unknown.Maya caps admission at funded slots. Leo stops unsafe actions. Noor schedules Sana; Imani checks forecast charges. Existing hours fund current checks.At the next weekly planning check, Noor brings a due-day slot plan and separate calibration hour; Leo brings release and stop records. Otherwise keep the cap.
Retire: Maya, product ownerRetire: Maya, product ownerRecurrent errors, open case IDs and cost of a manual route must be assessed; manual workload is uncertain.Maya ends admission. Leo reconciles open action IDs. Noor agrees and funds a staffed manual schedule, not the current correction allowance.End agent intake only after open writes are reconciled and a funded manual plan meets the same deadlines. Check service results in the next weekly meeting.

Take one decision to the meeting

Bring the cohort counts, separate cash and staff totals and Sana's dated review windows. Keep wider B intake on hold. Maya may fund only the requests Noor can staff before their deadlines. At the next weekly meeting, ask Leo for the checked stop and release records and Imani for the revised budget. Expand only after the longer review schedule and qualified decisions can be supported.

Context Window dispatch

Practical AI engineering you can actually use

Working code, experiments, and production lessons. Published when there is something worth sending, never padded with AI news.

Confirmation is required. Read the privacy policy. Unsubscribe at any time.

The email edition is preparing to launch. No campaigns are being sent yet.