Treat timed-out agent writes as unknown, not failed

The write can finish before the timeout

An agent asks a business service to issue a refund. The service saves the refund but its reply never reaches the agent's worker. The worker sees a timeout.

Sending the refund again may create a second refund. Reporting failure may also be wrong because the first refund already exists.

The same problem appears when a worker receives a receipt, then crashes before saving it. A receipt is the service's durable confirmation of a specific completed action. After restart, the worker has no saved proof of completion.

Put a durable operation ledger around the write tool. This is a database record that survives worker restarts. Save the action's identity before sending the request and keep uncertain outcomes in an explicit unknown state.

The service that receives the write is the downstream service. Recovery depends on what that service can prove or safely repeat. Local records alone cannot establish whether a remote action completed.

Follow one refund through a lost reply

Illustrative example: tenant north-shop requests a USD 48.50 refund for order order-731. The workflow assigns operation refund-204 before calling the refund tool. The downstream service records receipt receipt-882.

The service commits the refund, meaning it saves the change durably. Then the connection closes before the worker receives the receipt. The local operation becomes unknown while the refund still exists downstream.

Keep refund-204 when the agent tries again, even if the agent generates a new tool-call identifier. That identifier describes one invocation. The operation identifier describes the business action across all invocations.

If the service can safely replay the original request, use the same operation key. If it can look up the original receipt, retrieve that receipt. Otherwise leave refund-204 pending without sending another refund.

An unknown result is not permission to create refund-205. A second operation needs a separate intentional action, not a new name for the unresolved refund.

Save the operation before network activity

The local ledger and the downstream service cannot share a transaction. A transaction saves related database changes together or saves none of them. Here, the two systems save their changes separately.

Use this order: save prepared, save dispatching, send the write, then save the receipt with succeeded. Prepared means no dispatch has been authorized. Dispatching means the request may have left the worker.

The illustrative diagram separates the downstream commit from the local receipt save. A crash anywhere between dispatch authorization and receipt persistence needs conservative recovery.

Make state changes control dispatch

Store the tenant, operation identifier, tool name and exact business arguments in each ledger record. Also store its state, original downstream key, receipt and recovery history. Record the downstream capability used for recovery.

Enforce one row per tenant and operation identifier. Create or load that row before dispatch. Compare the saved arguments on every invocation, including invocations that load a completed operation.

Reject the request if an existing identifier arrives with changed arguments. For the refund, compare the order, amount and currency as well as the tool. Keep a stable argument format so property order cannot change the comparison.

Authorize the first send through one atomic database update from prepared to dispatching. Atomic means competing workers cannot both win that update. Only the worker that changed the row may make the initial send.

Commit that update before network activity. If its database result is uncertain, read the durable record before doing anything else. Do not send because an in-memory variable still says prepared.

After a timeout, save unknown when possible. A crash may prevent that save. Restart recovery must therefore treat abandoned dispatching records as unknown too.

Save the verified receipt and succeeded in one local transaction. Return success only after that save completes. A later invocation can then return the saved receipt without contacting the downstream service.

Choose recovery from the service's capabilities

Idempotency means repeating the same logical action without doing the business work twice. An idempotency key lets the downstream service recognize those repetitions. Recovery needs the original key and unchanged arguments.

For this recovery design, durable idempotency must cover concurrent attempts and survive downstream restarts. It must also return the original receipt. Merely rejecting a duplicate request does not establish success locally.

An authoritative lookup returns a reliable record of the operation by its stable identity. A matching completed record can recover the receipt without repeating the write.

Some lookup systems update later than the write system. This is eventual consistency. A temporary lookup miss in such a system does not prove that the write failed.

Use the table as an illustrative adapter contract. An adapter is the code that connects the executor to a specific business service. Its capability declaration controls which recovery actions are allowed.

A missing receipt is not a new dispatch permission

A service may keep idempotency keys only for a limited time. Store that deadline with the operation. Do not treat an expired key as safe because the local ledger still remembers it.

The adapter must account for requests still running when protection expires. Checking the local clock before sending is not enough if a delayed request can arrive after the downstream record disappears.

When safe replay cannot be established, prefer authoritative lookup. If lookup cannot establish completion, hold the operation. Never switch to a fresh key after a replay timeout.

Even authoritative absence needs care. It may establish that no write exists now while an earlier request can still arrive later. It does not by itself establish that the original request can never commit.

This design keeps an unknown operation pending after a lookup miss. Any path that permits another write needs proof that the earlier request cannot commit. An empty search result does not provide that proof.

Count downstream requests separately from business mutations. A mutation is a saved business change, such as one refund. Same-key replay may send another request while still producing only one mutation.

Keep business identity outside the model

Create the operation identifier in the application workflow before the tool runs. Persist its link to the authorized business action. Do not ask the model to generate or remember it.

On restart, load that link from durable storage. When the agent replans, attach the existing identifier to another attempt at the same action. An unresolved action should remain visible to the workflow.

Do not derive identity only from argument equality. A customer may intentionally receive two USD 48.50 refunds for different approved reasons. Those actions need different operation identifiers even when their tool arguments match.

The application must distinguish a new approved action from another attempt at an old one. Require an explicit workflow step for the new action. An executor cannot detect that mistake if the workflow silently supplies a fresh identifier.

Bind identity to the tenant and verify access on every call. Knowing an operation identifier must not let another tenant read its receipt or trigger its recovery.

Return pending as a normal tool result

A timeout exception often reaches retry code before the agent sees it. Keep write recovery inside the durable executor instead. Disable automatic write retries that bypass its operation record or change its downstream key.

Return a structured pending result while the outcome is unknown. Include the operation identifier and a clear next action. A read-only status check can be retried without repeating the refund.

The following JSON uses the illustrative refund values. It gives the agent no receipt and no permission to send another write.

1{
2  "operationId": "refund-204",
3  "status": "pending",
4  "receipt": null,
5  "nextAction": "check_status",
6  "message": "The refund outcome is unknown. Do not submit it again."
7}

Make the recovery decision explicit in C#

The application should show pending directly to the user. A model instruction alone is not a completion check. Only a stored receipt should permit a completed refund message or later work that depends on completion.

The illustrative RecoveryPolicy.Choose method separates recovery decisions from network calls. It chooses among a saved receipt, lookup, protected replay and a hold. It never authorizes a fresh write.

The adapter sets RecoveryFacts.CanReplaySafely only when the original key, unchanged arguments and downstream protection permit replay. Set RecoveryFacts.HasReceiptLookup only for authoritative lookup by operation identity.

The executor still owns database transactions and dispatch control. It must validate a recovered receipt before saving success. A pending lookup or another timeout leaves the operation unknown.

1namespace ContextWindow;
2

3public enum RecoveryAction
4{
5    ReturnStoredReceipt,
6    LookupOriginalReceipt,
7    ReplayOriginalRequest,
8    Hold
9}
10

11public sealed record RecoveryFacts(
12    bool HasStoredReceipt,
13    bool HasReceiptLookup,
14    bool CanReplaySafely);
15

16public static class RecoveryPolicy
17{
18    public static RecoveryAction Choose(RecoveryFacts facts)
19    {
20        if (facts.HasStoredReceipt)
21            return RecoveryAction.ReturnStoredReceipt;
22

23        if (facts.HasReceiptLookup)
24            return RecoveryAction.LookupOriginalReceipt;
25

26        if (facts.CanReplaySafely)
27            return RecoveryAction.ReplayOriginalRequest;
28

29        return RecoveryAction.Hold;
30    }
31}

Test the commit and reply as separate events

Build a local fault fixture, a test system that stops execution at chosen points. Use a .NET executor, a PostgreSQL ledger and a fake downstream service. Drive the same action through a deterministic workflow without a live model.

Give the fake service separate business data and transaction ownership. It must save the mutation and receipt before it drops the response. Dropping the request before commit tests a different failure.

Implement three service modes: durable idempotency, authoritative receipt lookup and neither capability. Keep the same business action and failure point across modes. The mode without either capability checks whether local bookkeeping causes an unsafe retry.

Compare a conventional executor with the protected executor. The conventional executor retries with a fresh downstream identity after the lost reply. Use the same commit-then-drop behavior for both.

Inspect the downstream database rather than relying on log messages. Record mutation counts, original receipts, local states and agent-facing results. Also count sends to catch hidden retry middleware.

The acceptance rule is strict. Fail the test if protected recovery repeats the business mutation or reports success without a matching receipt. For supported recovery after commit, also require the original receipt rather than an indefinite pending result.

Test identity and restart behavior too

For the conventional executor, assert that fresh-key retry exposes the duplicate under commit-and-response-loss. For the protected executor, assert the capability-specific outcomes under that same failure.

Kill the worker process at the chosen point and restart it with its old in-memory state gone. Keep both databases intact. Throwing an exception inside a running worker does not test that restart boundary.

Make two workers compete for the prepared operation. Verify that only one gains initial dispatch permission. Then pause the winning worker and start recovery to test late requests and late receipts.

Repeat recovery while the service is slow. Check that unknown never becomes a fresh dispatch permission. Expire idempotency protection and verify that the executor stops replaying.

Send changed arguments under the original identifier and require rejection before network activity. Send two intentionally separate actions with equal arguments and different identifiers. Both must remain possible.

Return a receipt for the wrong operation and require rejection. Simulate a delayed lookup miss and require pending. Test a local database outage after remote success as another path into unknown.

For repeatable runs, pin dependencies and container images. Provide container startup, a reset limited to fixture data and one test entry point. Keep reset operations separate from crash recovery, which must preserve existing records.

Measure actual test duration and resource use during execution. This is a correctness comparison, not a throughput benchmark. It needs no model-token spend or paid downstream service.

Assign an owner to pending work

The team that owns the write-tool adapter should own its capability declaration and recovery tests. Review them when downstream retention, lookup behavior, retry middleware or workflow identity handling changes.

Give operations staff a queue of unresolved actions. Show the tenant, operation identifier, saved arguments, first dispatch time and recovery history. Include the protection deadline and the person responsible for reconciliation.

Reconciliation means checking business records to establish what happened. A manual retry button must not bypass the same recovery rules. Require a matching receipt before closing an operation as succeeded.

Do not issue a compensating action merely because a timeout occurred. A compensation is another business write intended to counter an earlier one. It needs a known target and its own durable operation identity.

A tool without downstream recovery support can remain pending indefinitely. That is the cost of refusing a duplicate when the original result cannot be established. Track hold age so unresolved work does not disappear from view.

Start Monday with one consequential write tool. Persist its operation identity, make pending visible and stop the worker immediately after the downstream commit. Require recovery to return the original receipt or keep the action pending without repeating it.

Source

The complete sample is available at github.com/alex-janjic/ContextWindow.Agents.Unknownoutcomes.