Conversations

Pass a conversation id to a run, and Typeshift stores every step under it. The next run picks up where the last one ended, even after a crash.

Tracking a conversation

var conversationId = 7;

await support.RunAsync("Where is my refund?", conversationId: conversationId);
await support.RunAsync("When will it arrive?", conversationId: conversationId);

^{snippet source | anchor}

The first run starts the conversation and the second resumes it. Typeshift puts the stored messages ahead of the new input, so the model sees the whole conversation. Because Typeshift sends the history itself, it never passes the conversation id on to the provider. Settings ids and conversation ids are independent, and a run may pass either or both. A conversation does not remember a settings id, so pass it to every run of the conversation that should use the stored settings, the run that answers an approval included.

Typeshift never creates ids (the id type). You choose the conversation id, for example the key of your own record of the conversation, and pass the same id to every run of that conversation.

Check the caller before you pass a conversation id

Typeshift resumes whichever conversation it is given, so deciding who may see one is your app's job. Check the caller against your own record of the conversation, as you would for any other id from a request.

What is stored

A conversation is a list of entries, appended as they happen: each run's input, every model response with its model and token usage, the tool results of each round of calls, and an entry that ends the run and records its mode. Your store receives each entry through AppendConversationAsync (Implementing a store).

Each entry is a ConversationEntry: the name of its agent in Agent, its Messages and the time it was stored in At, plus the Model and Usage of a model response, and EndsRun on the entry that ends a run. IsSummary is true on the entry compaction appends when it summarizes, which is how your code and your store tell a summary apart (Start from the last summary).

An entry serializes itself with Serialize() and ConversationEntry.Deserialize(json), and JSON libraries handle it as it is. System.Text.Json writes and reads that same JSON, and Newtonsoft.Json writes it as a string, so a store or an API can pass entries on with either library and no settings.

var store = provider.GetRequiredService<IConversationStore>();
var entries = await store.LoadConversationAsync(conversationId, ct);

Console.WriteLine($"{entries.Count} entries, {entries.Sum(entry => entry.Usage?.TotalTokenCount)} tokens");

^{snippet source | anchor}

Runs that fail or stop early are stored too:

• A failed run stores an entry with an ErrorContent that ends the run, then rethrows the exception to your code.
• A streaming run that fails, is cancelled, or that your code stops reading early stores the updates that had arrived.
• A cancelled run is stored too, because Typeshift stores without the run's cancellation token.
• A run cut off by a crash has no ending entry, and resuming continues it.

When the store fails to save, the failure is logged and recorded on the run's span, and the run still returns its answer. Whenever a run loads the conversation, it leaves tool calls that never got a result and error messages out of what the model sees, because providers reject them.

Some providers, such as local servers with an OpenAI-compatible API, number their tool calls per response, so a later call can get an id the conversation holds already. Typeshift renames such a call, as call_1_2, before it stores or runs it, so the stored entries, ApprovalRequests and the tool's result all carry the new id.

Resuming

Each model response is stored as it arrives, and tool results before the next model call, so an app that crashes mid-run loses at most the calls in flight. ResumeAsync continues the conversation's interrupted run from its last stored step, without new input:

var resumed = await support.ResumeAsync(interrupted, settingsId);

^{snippet source | anchor}

On a conversation whose last run ended, ResumeAsync throws an InvalidOperationException. After a restart, the entries tell you which conversations to resume: when the last entry that LoadConversationAsync returns has EndsRun false, a crash cut that run off. A new run with input works on an interrupted conversation too: it starts from the interrupted run's steps and adds its own input.

A tool can be called twice

A call interrupted while its tool ran is left out on resume. The model decides again and may call the tool a second time, so make tools with side effects safe to repeat.

One run at a time

Within a process, a conversation has one run at a time. Starting a second while the first goes on throws an InvalidOperationException, so steer the running one or wait until it ends. A run that starts while another is still loading the conversation waits until that run has begun. Across processes, keeping to one run per conversation is up to your app.

Live runs

ConversationTracker lists the runs with a conversation id that are going on in this process, each with its conversation, the agent that answers, its model and when it started:

var tracker = provider.GetRequiredService<ConversationTracker>();

foreach (var run in tracker.Runs)
    Console.WriteLine($"{run.Agent} is answering conversation {run.Conversation} on {run.Model}");

^{snippet source | anchor}

WatchRunsAsync tells a UI when that list changes, so a list of running conversations needs no polling. It yields the runs going on, then each run as it starts and again as it ends, with Ended set, until you cancel its token. Cancelling ends the loop with an OperationCanceledException, so catch it where you stop watching. It works without WithRealtimeTracking():

await foreach (var run in tracker.WatchRunsAsync(ct))
    Console.WriteLine($"Conversation {run.Conversation} is {(run.Ended ? "idle" : "running")}");

^{snippet source | anchor}

Steering

TrySteer hands new input to a conversation's run while it works, for example when a user adds a request halfway through. The model receives it before its next call, and the run does not end before then. The input is stored like any other. When the run had asked for approval, its requests close once it answers the input, as they would for a new message (When a request closes).

TrySteer returns false when no run of the conversation is going on in this process, or when that run is already ending. Start a run with the input instead:

var input = "Send me the invoice too.";

if (!tracker.TrySteer(conversationId, input))
    await support.RunAsync(input, settingsId, conversationId);

^{snippet source | anchor}

To stop a run, cancel its token. It is stored up to where it stopped.

Live updates

With WithRealtimeTracking(), Typeshift publishes the updates of runs as they happen, and WatchAsync reads one conversation's updates until you cancel its token, which ends the loop with an OperationCanceledException, as with WatchRunsAsync. A streaming run publishes its updates as they stream; a run that does not stream publishes each model response and tool result as it stores it. A UI can forward them over SignalR, for example:

await foreach (var update in tracker.WatchAsync(conversationId, ct))
    Console.Write(update.Text);

^{snippet source | anchor}

Text, reasoning and usage updates are merged and published at most every 100 ms, and other updates go out at once. A run never waits for a watcher, so a watcher that falls more than 256 updates behind loses the oldest ones. You can change both limits with RealtimeTrackingOptions:

services.AddTypeshift()
    .WithRealtimeTracking(new RealtimeTrackingOptions
    {
        PublishInterval = TimeSpan.FromMilliseconds(250),
        MaxPendingUpdates = 1024
    });

^{snippet source | anchor}

Without WithRealtimeTracking(), WatchAsync throws an InvalidOperationException.

Live updates stay in their process

A watcher sees only the runs of its own process. When your app runs in several processes, watch in the process that runs the conversation.

Compaction

A coding agent can make hundreds of model calls in one run, and its history soon outgrows the model's context. Compaction shrinks the history each call sees. It is off until WithMaxInputTokens sets a limit, and the other run options let such a run go on for long enough: here WithMaxToolIterations(500) raises the limit on rounds of tool calls to 500. The agent also starts in plan mode, which has nothing to do with compaction:

WithRunOptions()
    .WithMode(AgentMode.Plan)
    .WithMaxToolIterations(500)
    .WithMaxInputTokens(150_000)
    .WithSummaryModel("claude-haiku-5-5")
    .Configurable(r => r.Mode)
    .Configurable(r => r.MaxToolIterations)
    .Configurable(r => r.SummaryModel);

^{snippet source | anchor}

Before every model call, Typeshift estimates the call's input tokens at about four characters per token, corrected by what the provider reported for the run's last call, or for the conversation's last call before the run. When the estimate reaches the maximum, it shrinks the history to WithTargetInputTokens, half the maximum unless set:

1. It clears the results of the oldest tool calls the model has already read. It keeps the calls, so the model knows what it did and can call again.
2. If that is not enough, it summarizes the oldest turns with the summary model, the run's own unless you name another.

The instructions and the latest call of a tool named write_todos, if the agent offers one, always stay, and a cut never separates a tool call from its result. Every later model call of the same run starts from the history as compaction left it, with the same results cleared and the same summary.

The stored conversation keeps everything. Compaction appends an entry of its own, with IsSummary set to true, that holds the summary and the messages it kept after it. Later runs start from that entry instead of summarizing again, and read nothing before it: the entry also keeps the conversation's mode, its approvals for the conversation and the background agents still working.

Related

• Running agents: passing a conversation id to a run.
• How a run works: where a run resumes and stores its steps.
• Implementing a store: the store methods that keep conversations.
• Builders: the run options of compaction and their defaults.
• Models and providers: which model summarizes, and its provider.
• Multiple agents: handoffs, which need a conversation.
• Serving a chat endpoint: a conversation per chat, behind ASP.NET Core endpoints.