You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
.NET: Allow persisting partial interaction when streaming is cancelled - #9790
When ChatClientAgent.RunStreamingAsync is used with an AgentSession and the caller cancels the CancellationToken mid-stream, the run is treated as a failure: providers are notified with InvokeException set and nothing is stored. Neither the user request nor the partial assistant response ends up in the session.
In chat UIs, a "Stop generating" button usually means stop now, but keep what the user already saw. With the current behavior, the conversation shown in the UI diverges from the one stored in the session, and on the next turn the model knows neither the last user message nor the partial answer. Today this can only be worked around outside the framework, and even then the partial response updates are not reachable, because they only exist inside ChatClientAgent.
This PR adds an opt-in way to persist the interaction up to the point of cancellation, implementing the .NET part of the proposal discussed in #8157.
Description & Review Guide
What are the major changes?
New experimental (MAAI001) opt-in setting, available at two levels:
ChatClientAgentOptions.PersistPartialMessagesOnCancellation (bool, default false): agent-level default, copied by Clone().
ChatClientAgentRunOptions.PersistPartialMessagesOnCancellation (bool?, default null): per-run override (e.g. "Stop" keeps the partial answer, "Regenerate" discards it), copied by Clone().
New public constant ChatClientAgent.PartialResponseMarkerKey ("agent_framework.partial_response"). Every persisted partial response message gets this key set to true in ChatMessage.AdditionalProperties, so providers, UIs and evaluators can recognize truncated responses without a new InvokedContext API.
ChatClientAgent.RunCoreStreamingAsync: when MoveNextAsync throws an OperationCanceledException, the caller's token is cancelled and the setting is enabled, the request messages and the sanitized partial response are sent to the ChatHistoryProvider and the AIContextProviders through the regular success channel (NotifyProvidersOfNewMessagesAsync). The original exception is then rethrown.
Sanitization of the partial response (CreatePartialResponseMessages), so that the stored history stays valid for the next service call: tool calls without a matching tool result, tool results without a matching tool call, pending input requests (e.g. ToolApprovalRequestContent) and empty text are removed; messages left without content are dropped.
PerServiceCallChatHistoryPersistingChatClient: when RequirePerServiceCallChatHistoryPersistence is enabled, the decorator applies the same behavior to the current service call. Earlier service calls of the same run are already persisted, and ChatClientAgent keeps skipping end-of-run persistence, so nothing is written twice.
New warning log entry LogAgentChatClientPartialMessagesNotificationFailed.
Unit tests: new ChatClientAgent_PartialPersistenceOnCancellationTests (flag on/off, cancellation before the first update, per-run overrides in both directions, unmatched and matched tool calls, non-caller cancellation, provider failure, per-service-call persistence with and without the flag, background responses, service-managed history, next run receiving the persisted partial turn), plus Clone() coverage for both option classes.
What is the impact of these changes?
No behavior change by default: the setting is opt-in and existing runs keep today's atomic behavior.
Scope: streaming runs with framework-managed chat history only. The setting is ignored for non-streaming runs (no partial content is exposed), for service-managed history (ConversationId), and for background responses or continuation tokens (the caller can resume the run instead). Cancellations that do not come from the caller's token (e.g. an HttpClient timeout surfacing as TaskCanceledException) still go through the failure path.
The caller always observes the original OperationCanceledException, rethrown with throw;. The notification uses CancellationToken.None, because the caller's token is already cancelled, and any failure while notifying providers is logged and swallowed, so it cannot mask the original exception. When the caller's catch runs, the session already contains the partial turn.
Existing ChatHistoryProvider and AIContextProvider implementations (InMemory, Cosmos, Valkey, custom) work without changes.
What do you want reviewers to focus on?
No double notification (point 7 of the proposal). When the partial messages notification is sent, the failure notification for the same run is not sent: the two paths are mutually exclusive. This is why each MoveNextAsync call site (two in ChatClientAgent, two in PerServiceCallChatHistoryPersistingChatClient) now has two separate catch blocks: the first one uses an exception filter (when (ShouldNotifyProvidersOfPartialMessages…)) and sends the partial messages through the success channel, while the second one is the existing failure path, unchanged. The filter methods are pure checks, since they run before the stack is unwound.
Early disposal of the stream is not covered. If the consumer exits the await foreach loop with break (or disposes the enumerator) without cancelling the token, nothing is persisted. This is still an open question in the proposal: it is a common "Stop" pattern in UIs, but ToolApprovalAgent also relies on yield break, so it may need a different trigger or a separate flag.
Per-service-call persistence and cancellation during tool execution. With RequirePerServiceCallChatHistoryPersistence enabled, if the cancellation happens while a tool is running, the tool call remains in the history without a result, because it was already persisted by the previous service call. This also happens with the current code and is not changed by this PR.
Open points from the proposal that would benefit from maintainer input: the name of the marker key and whether it should stay a public constant, and whether AIContextProviders should also be notified of partial responses (they currently are, consistently with chat history).
This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.
Adds an opt-in PersistPartialMessagesOnCancellation setting on ChatClientAgentOptions (agent-level) and ChatClientAgentRunOptions (per-run override). When a streaming run is cancelled by the caller's token, the request messages and the sanitized partial response are persisted through the success channel, marked with ChatClientAgent.PartialResponseMarkerKey. Works with both end-of-run and per-service-call persistence without duplicate writes; the original OperationCanceledException is always rethrown.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Both persistence paths ignore continuation tokens received in response updates, allowing cancelled partial turns to be saved again on resume and duplicating requests and partial responses in history.
Adds opt-in partial-turn persistence to .NET ChatClientAgent, addressing #8157 so cancelled streaming responses can remain in framework-managed conversation history.
Changes:
Adds experimental agent-level and per-run options, plus a marker for persisted partial responses.
Adds cancellation handling, content sanitization, and provider-failure logging for end-of-run and per-service-call persistence.
Adds cancellation behavior tests, option-cloning coverage, and public API baselines.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
After cancellation with partial persistence, approval decorators retain retry state for completed tool calls, allowing duplicate results or repeated tool execution on the next turn.
…ersisted on cancellation
When a cancelled streaming run persists its partial interaction, remove from the
session the pending approval requests and stored auto-approved calls whose results
are now in the chat history, keeping unfinished calls for retry. With per-service-call
persistence the next run handles approvals before the history is loaded, so leftover
state would settle or execute the same calls again.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A context-provider notification failure after history persistence can leave settled tool-approval retry state uncleared, causing duplicate tool execution or rejection results on the next turn.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Reconcile tool approval state after history persistence
If an AIContextProvider notification throws after chat-history persistence succeeds, this await skips RemoveSettledToolApprovalState even though the tool results are already stored. With per-service-call persistence, the retained retry state can cause the next turn to execute an auto-approved tool again or append a duplicate rejection result. Reconcile settled approval state immediately after successful history persistence, independently of context-provider notifications, and add a cancellation → context-provider failure → next-turn regression test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Actions
Inactive
Motivation & Context
When
ChatClientAgent.RunStreamingAsyncis used with anAgentSessionand the caller cancels theCancellationTokenmid-stream, the run is treated as a failure: providers are notified withInvokeExceptionset and nothing is stored. Neither the user request nor the partial assistant response ends up in the session.In chat UIs, a "Stop generating" button usually means stop now, but keep what the user already saw. With the current behavior, the conversation shown in the UI diverges from the one stored in the session, and on the next turn the model knows neither the last user message nor the partial answer. Today this can only be worked around outside the framework, and even then the partial response updates are not reachable, because they only exist inside
ChatClientAgent.This PR adds an opt-in way to persist the interaction up to the point of cancellation, implementing the .NET part of the proposal discussed in #8157.
Description & Review Guide
What are the major changes?
MAAI001) opt-in setting, available at two levels:ChatClientAgentOptions.PersistPartialMessagesOnCancellation(bool, defaultfalse): agent-level default, copied byClone().ChatClientAgentRunOptions.PersistPartialMessagesOnCancellation(bool?, defaultnull): per-run override (e.g. "Stop" keeps the partial answer, "Regenerate" discards it), copied byClone().ChatClientAgent.PartialResponseMarkerKey("agent_framework.partial_response"). Every persisted partial response message gets this key set totrueinChatMessage.AdditionalProperties, so providers, UIs and evaluators can recognize truncated responses without a newInvokedContextAPI.ChatClientAgent.RunCoreStreamingAsync: whenMoveNextAsyncthrows anOperationCanceledException, the caller's token is cancelled and the setting is enabled, the request messages and the sanitized partial response are sent to theChatHistoryProviderand theAIContextProviders through the regular success channel (NotifyProvidersOfNewMessagesAsync). The original exception is then rethrown.CreatePartialResponseMessages), so that the stored history stays valid for the next service call: tool calls without a matching tool result, tool results without a matching tool call, pending input requests (e.g.ToolApprovalRequestContent) and empty text are removed; messages left without content are dropped.PerServiceCallChatHistoryPersistingChatClient: whenRequirePerServiceCallChatHistoryPersistenceis enabled, the decorator applies the same behavior to the current service call. Earlier service calls of the same run are already persisted, andChatClientAgentkeeps skipping end-of-run persistence, so nothing is written twice.LogAgentChatClientPartialMessagesNotificationFailed.ChatClientAgent_PartialPersistenceOnCancellationTests(flag on/off, cancellation before the first update, per-run overrides in both directions, unmatched and matched tool calls, non-caller cancellation, provider failure, per-service-call persistence with and without the flag, background responses, service-managed history, next run receiving the persisted partial turn), plusClone()coverage for both option classes.What is the impact of these changes?
ConversationId), and for background responses or continuation tokens (the caller can resume the run instead). Cancellations that do not come from the caller's token (e.g. anHttpClienttimeout surfacing asTaskCanceledException) still go through the failure path.OperationCanceledException, rethrown withthrow;. The notification usesCancellationToken.None, because the caller's token is already cancelled, and any failure while notifying providers is logged and swallowed, so it cannot mask the original exception. When the caller'scatchruns, the session already contains the partial turn.ChatHistoryProviderandAIContextProviderimplementations (InMemory, Cosmos, Valkey, custom) work without changes.What do you want reviewers to focus on?
MoveNextAsynccall site (two inChatClientAgent, two inPerServiceCallChatHistoryPersistingChatClient) now has two separatecatchblocks: the first one uses an exception filter (when (ShouldNotifyProvidersOfPartialMessages…)) and sends the partial messages through the success channel, while the second one is the existing failure path, unchanged. The filter methods are pure checks, since they run before the stack is unwound.await foreachloop withbreak(or disposes the enumerator) without cancelling the token, nothing is persisted. This is still an open question in the proposal: it is a common "Stop" pattern in UIs, butToolApprovalAgentalso relies onyield break, so it may need a different trigger or a separate flag.RequirePerServiceCallChatHistoryPersistenceenabled, if the cancellation happens while a tool is running, the tool call remains in the history without a result, because it was already persisted by the previous service call. This also happens with the current code and is not changed by this PR.AIContextProviders should also be notified of partial responses (they currently are, consistently with chat history).Related Issue
Fixes #8157
AI Assistance
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.