Make agent tool retries idempotent before enabling automatic recovery

A timeout does not mean the action failed

An agent calls a tool to create a ticket. The ticket is created, but the reply is lost before the agent receives it. The agent sees a timeout, while the ticket service sees a finished request. If the agent tries again, it creates a second ticket.

Engineers call this idempotency. It means that repeating one action returns the first result instead of doing the work twice. Telling the model to use the same details is not enough. A retry may have a new call ID or slightly different JSON. Your application must decide which calls represent the same action.

This tutorial tests one failure point. The outside system finishes the action and the application saves the result, but the reply never reaches the agent. A retry with the same action key can return the saved result. This method cannot solve a different case where the outside action may have finished but no result was saved.

Let the application choose the action key

Create the key when your product defines the action. Examples include one refund for one order or one release sent to one environment. Use the same key for network retries and new calls from the model. Do not rely on a call ID made by the model. Do not rely only on a hash of the model's JSON, because the same request can be written in different ways.

Save an intent fingerprint with the key. This fingerprint is a stable value based only on the fields that define the approved action. Reject the key if those fields change. This stops an old key from returning a result for a different amount, person, environment, or resource. Product code must decide which fields matter.

Approval and retry safety solve different problems. Approval says whether someone allowed the exact action. The execution record says whether that action already ran. Use both when your system has both risks.

Check the saved result before calling the tool

The tool adapter gets a LogicalActionKey from trusted application context. The model must not choose this key. First, call IExecutionLedger.ClaimAsync(). Only a new claim may call IMutatingTool.ExecuteAsync(). If the claim is complete, return its saved ToolOutcome.

The code below shows the tested failure case. IFaultInjector.AfterLedgerCommitAsync() runs after IExecutionLedger.CompleteAsync() so the test can lose the first reply without deleting the saved result. The full sample and its tests are in the linked GitHub repository.

1namespace ContextWindow;
2

3public sealed record LogicalActionKey(string TenantId, string ActionId);
4

5public sealed record ToolRequest(string IntentFingerprint, string Payload);
6

7public sealed record ToolOutcome(string ExternalId, string ResultJson);
8

9public enum ClaimState
10{
11    Acquired,
12    InProgress,
13    Completed,
14    SafelyFailed,
15    OutcomeUnknown,
16    IntentConflict
17}
18

19public sealed record ExecutionClaim(
20    ClaimState State,
21    long Fence,
22    ToolOutcome? Outcome);
23

24public interface IExecutionLedger
25{
26    ValueTask<ExecutionClaim> ClaimAsync(
27        LogicalActionKey key,
28        string intentFingerprint,
29        CancellationToken cancellationToken);
30

31    ValueTask CompleteAsync(
32        LogicalActionKey key,
33        long fence,
34        ToolOutcome outcome,
35        CancellationToken cancellationToken);
36}
37

38public interface IMutatingTool
39{
40    ValueTask<ToolOutcome> ExecuteAsync(
41        ToolRequest request,
42        string downstreamIdempotencyKey,
43        CancellationToken cancellationToken);
44}
45

46public interface IFaultInjector
47{
48    ValueTask AfterLedgerCommitAsync(CancellationToken cancellationToken);
49}
50

51public sealed class IdempotentToolExecutor(
52    IExecutionLedger ledger,
53    IMutatingTool tool,
54    IFaultInjector faultInjector)
55{
56    public async ValueTask<ToolOutcome> ExecuteAsync(
57        LogicalActionKey key,
58        ToolRequest request,
59        CancellationToken cancellationToken)
60    {
61        ExecutionClaim claim = await ledger.ClaimAsync(
62            key,
63            request.IntentFingerprint,
64            cancellationToken);
65

66        if (claim.State is ClaimState.Completed)
67        {
68            return claim.Outcome
69                ?? throw new InvalidOperationException(
70                    "A completed execution must contain an outcome.");
71        }
72

73        if (claim.State is not ClaimState.Acquired)
74        {
75            throw new InvalidOperationException(
76                $"Logical action cannot execute while it is {claim.State}.");
77        }
78

79        ToolOutcome outcome = await tool.ExecuteAsync(
80            request,
81            $"{key.TenantId}:{key.ActionId}",
82            cancellationToken);
83

84        using var completionTimeout = new CancellationTokenSource(
85            TimeSpan.FromSeconds(5));
86

87        await ledger.CompleteAsync(
88            key,
89            claim.Fence,
90            outcome,
91            completionTimeout.Token);
92

93        await faultInjector.AfterLedgerCommitAsync(cancellationToken);
94        return outcome;
95    }
96}

Test the gap between saving and replying

The test uses a database-backed execution record and a fake outside system that counts calls. The first attempt calls the outside system once, gets result R1, and saves R1. IFaultInjector.AfterLedgerCommitAsync() then throws a timeout before the caller receives R1. The second attempt uses the same LogicalActionKey and the same approved action. A third attempt uses a different model call ID and a different JSON field order, but it still describes the same approved action.

The checks are exact: the outside action runs once, the execution record is Completed, and every retry returns R1. Another test starts two calls with the same key and checks that only the claim owner reaches IMutatingTool.ExecuteAsync(). A final test reuses the key for a different action and confirms that the old result is not returned.

The complete sample includes these tests. The commands below run the response-loss test and the two-call race test.

$ dotnet test tests/ContextWindow.Agents.IdempotentTools.Tests/ContextWindow.Agents.IdempotentTools.Tests.csproj --configuration Release --filter FullyQualifiedName~CommitThenTimeout
This test loses the first reply after the result is saved, then checks that the retry returns the same result.
$ dotnet test tests/ContextWindow.Agents.IdempotentTools.Tests/ContextWindow.Agents.IdempotentTools.Tests.csproj --configuration Release --filter FullyQualifiedName~ConcurrentDuplicate
This test checks that two calls with the same key produce one outside action.

How to handle each saved state

For Completed, return the saved ToolOutcome and record that it was reused. If InProgress has an active lease, wait or tell the caller to try later. For SafelyFailed, retry only when the system can prove that the action did not happen. For OutcomeUnknown, stop automatic calls and check the outside system.

Use the same provider idempotency key or a provider lookup tied to a stable business reference. If the check finds the action, save its result as Completed and return it. If the check proves that no action happened, your rules may allow a new attempt. If the result is still unclear, leave the row in OutcomeUnknown for a person to review.

Turn on automatic recovery only after you define what happens in all four states. A retry setting only sends the request again. It does not identify the action, return a saved result, or explain an unknown outside result.

Start with one high-risk tool

Choose one tool where a duplicate would be costly. Define when two calls mean the same action and add a required LogicalActionKey to trusted application context. Add a table with a unique tenant and action key, an intent fingerprint, state, attempt version, timestamps, and a saved result or result reference.

Send every call through IdempotentToolExecutor.ExecuteAsync() before you enable automatic retries. Lose the first reply after IExecutionLedger.CompleteAsync(), then test the same request, a new model call, and two calls at the same time. Do not enable recovery until only one outside action runs and every retry gets the same result.

Next, test the case where the outside system may finish but the adapter does not receive its reply. Write down how the provider supports idempotency or result lookup before you add this check. If it supports neither, require a person to review the state instead of retrying.

What this design cannot solve

An execution record cannot make your database and an outside system act as one transaction. It can return a result it saved and stop two local callers from running the same action. It cannot discover an outside action that finished when no result was saved. For that, the outside system must support an idempotency key, a lookup, or another reliable check.

Each product must define what counts as one action. A key that is too narrow allows duplicates under new keys. A key that is too broad blocks separate work or returns the wrong result. The intent fingerprint detects changed details, but the product must still decide which fields matter.

How long you keep the record also matters. Keep the unique key longer than every possible retry. You may need to delete a private result sooner. In that case, keep a small record or result reference that still blocks the action from running twice.

Source

The complete sample is available at github.com/alex-janjic/ContextWindow.Agents.Idempotenttools.