Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/specs/004-python-function-calling-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,10 @@ that manually replay messages own the equivalent rule: do not resend an approval
original shape and fails explicitly when that provider representation is unavailable or inconsistent. Locally
generated shell outputs preserve stdout, stderr, truncation notices, non-zero exits, and timeout outcomes and are
sent in both service-side and stateless continuation modes.
A `shell_call` for a configured local executor with valid item/call identifiers and `commands=[]` is a no-op:
return the call paired with a `shell_call_output` containing `output=[]`, without entering local execution or
approval. Preserve `max_output_length` when supplied. This applies to explicit local environments and omitted
environments resolved to the configured local executor; hosted, unconfigured, and malformed calls are unchanged.

### Reasoning-bound calls

Expand Down Expand Up @@ -592,6 +596,7 @@ that manually replay messages own the equivalent rule: do not resend an approval
| Parallel calls | Results retain the corresponding call ids and execution count. | `test_max_function_calls_limits_parallel_invocations`, `test_streaming_multiple_function_calls_parallel_execution` |
| Sequential calls | Calls execute one at a time in model order, including after approval replay; mixed local/Host-owned siblings retain that order, approval-only batches wait for every decision before executing, each call has isolated context, and middleware termination takes effect after the accepted batch finishes. | `packages/core/tests/core/test_tools.py::test_sequential_function_invocation_runs_calls_in_model_order`, `test_sequential_function_invocation_isolates_context`, `test_sequential_function_invocation_finishes_batch_after_termination`, `packages/core/tests/core/test_harness_tool_approval.py::test_mixed_batch_hides_already_approved_request_until_approval_replay`, `test_sequential_approval_replay_preserves_model_order_when_responses_are_reversed`, `test_sequential_approval_replay_waits_for_the_complete_batch`, `packages/core/tests/core/test_function_invocation_logic.py::test_sequential_mixed_batch_preserves_approval_and_host_model_order`, `packages/core/tests/workflow/test_agent_executor_tool_calls.py::test_workflow_cancels_host_member_of_mixed_batch_with_terminal_result`, `test_workflow_host_cancellation_preserves_occurrence_with_reused_call_id` |
| Informational-only call | The call is returned but not executed or approved. | `test_informational_only_function_call_is_not_invoked`, `test_informational_only_function_call_does_not_request_approval`, `test_streaming_informational_only_function_call_is_not_invoked` |
| Empty local shell call | An empty command list for a configured local executor returns a paired empty shell output without execution or approval; both parsers preserve the pair and stateless replay preserves output limits. Hosted, unconfigured, and malformed calls do not receive synthetic results. | `packages/openai/tests/openai/test_openai_chat_client.py::test_empty_local_shell_call_replays_with_empty_output`, `test_empty_local_shell_call_parsing_emits_pair_without_execution`, `test_empty_shell_result_is_not_synthesized_for_hosted_unconfigured_or_malformed_calls` |
| OpenAI hosted/local shell boundary | Hosted shell calls remain informational in streaming and non-streaming responses even when a local executor is configured; only valid explicit local-shell items or shell calls marked with a local environment can execute, local execution preserves its configured approval mode, stateless loops preserve the complete provider shell transcript or fail explicitly, and locally generated shell outputs preserve stdout, stderr, truncation notices, non-zero exits, and timeout outcomes in every continuation mode. | `packages/openai/tests/openai/test_openai_chat_client.py::test_response_content_creation_with_shell_call_remains_hosted_with_local_tool`, `test_parse_chunk_from_openai_shell_call_done_remains_hosted`, `test_parse_chunk_from_openai_local_environment_shell_call_done_emits_command`, `test_mixed_shell_calls_only_invoke_explicit_local_shell_call`, `test_stateless_shell_transcript_without_provider_item_fails`, `test_prepare_messages_keeps_local_shell_output_under_storage`, `test_shell_output_payloads_use_structured_function_result_items`, `test_malformed_local_environment_shell_call_is_not_executable`, `test_response_content_creation_with_local_shell_call_maps_to_function_call`, `test_malformed_local_shell_call_is_not_executable`, `test_response_function_call_named_local_shell_is_informational`, `test_parse_chunk_function_call_named_local_shell_is_informational`, `test_local_shell_tool_requires_approval_before_function_loop_execution` |
| Declaration-only call | The call is surfaced as user input and is not executed; streaming arguments appear once while finalized request metadata remains available. | `test_declaration_only_tool`, `test_streaming_declaration_only_tool_preserves_metadata_without_duplicate_arguments` |
| Declaration-only mixed batch | Known local siblings are never converted into Host pauses. With an authoritative session they remain deferred until every Host pause occurrence is answered, survive session serialization with a process-portable logical-run budget (including a reclassified approval continuation), reclassify against the current tool policy, execute exactly once using the configured concurrent or sequential invocation mode, replay results in original model order, and remain inert on replay; without one they execute on the same turn under the same configured invocation mode. Streaming updates and finalization preserve one Host pause and one result per local sibling. | `test_mixed_declaration_only_batch_defers_siblings_until_all_host_results`, `test_mixed_declaration_only_deferred_call_keeps_ordinary_middleware_semantics`, `test_mixed_declaration_only_deferred_call_reclassifies_changed_approval_policy`, `test_mixed_declaration_only_budget_survives_reclassification_to_approval`, `test_mixed_declaration_only_deferred_call_does_not_execute_removed_tool`, `test_mixed_declaration_only_budget_is_process_portable_without_approval_middleware`, `test_sessionless_mixed_declaration_only_batch_executes_siblings_sequentially`, `test_sessionless_mixed_declaration_only_batch_executes_siblings_concurrently`, `test_streaming_sessionless_mixed_declaration_only_batch_returns_pause_and_result` |
Expand Down
25 changes: 25 additions & 0 deletions python/packages/openai/agent_framework_openai/_chat_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -2596,6 +2596,21 @@ def _shell_item_to_contents(self, item: Any, local_shell_tool_name: str | None)
raw_representation=item,
)
)
if (
local_shell_tool_name
and is_local_environment
and shell_call_id
and shell_call_item_id
and isinstance(raw_shell_commands, list)
and not raw_shell_commands
):
contents.append(
Content.from_shell_tool_result(
call_id=shell_call_id,
outputs=[],
max_output_length=shell_max_output,
)
Comment on lines +2608 to +2612
)
elif item_type == "local_shell_call":
raw_local_call_id = getattr(item, "call_id", None)
local_call_id = raw_local_call_id if isinstance(raw_local_call_id, str) else ""
Expand Down Expand Up @@ -2853,6 +2868,16 @@ def _prepare_shell_transcript_item_for_openai(
expected_type: Literal["shell_call", "shell_call_output", "local_shell_call"],
) -> dict[str, Any]:
"""Restore a provider-issued shell transcript item for stateless replay."""
if (
expected_type == "shell_call_output"
and content.raw_representation is None
and content.call_id
and content.outputs == []
):
output: dict[str, Any] = {"type": "shell_call_output", "call_id": content.call_id, "output": []}
if content.max_output_length is not None:
output["max_output_length"] = content.max_output_length
return output
payload = cls._serialize_provider_payload(content.raw_representation)
if isinstance(payload, Mapping):
typed_payload = cast("Mapping[str, Any]", payload)
Expand Down
121 changes: 121 additions & 0 deletions python/packages/openai/tests/openai/test_openai_chat_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -2589,6 +2589,127 @@ def test_malformed_local_environment_shell_call_is_not_executable(
assert all(content.type != "function_call" for content in contents)


@pytest.mark.parametrize("max_output_length", [None, 4096])
@pytest.mark.parametrize("round_trip", [False, True])
@pytest.mark.parametrize("environment", [{"type": "local"}, None])
def test_empty_local_shell_call_replays_with_empty_output(
max_output_length: int | None, round_trip: bool, environment: dict[str, str] | None
) -> None:
"""Zero local commands must replay with a matching empty result, not an orphan call."""
client = OpenAIChatClient(model="test-model", api_key="test-key")
item = ResponseFunctionShellToolCall.model_validate({
"type": "shell_call",
"id": "sh_empty",
"call_id": "call_empty",
"status": "completed",
"action": {"commands": [], "max_output_length": max_output_length},
"environment": environment,
})

contents = client._shell_item_to_contents(item, "run_shell")

assert all(content.type != "function_call" for content in contents)
message = Message("assistant", contents)
if round_trip:
restored_result = Message.from_json(Message("assistant", [contents[1]]).to_json()).contents[0]
message = Message("assistant", [contents[0], restored_result])
replay = client._prepare_messages_for_openai(
[message, Message("user", "Continue")],
request_uses_service_side_storage=False,
)

shell_calls = [entry for entry in replay if entry["type"] == "shell_call"]
assert len(shell_calls) == 1
assert shell_calls[0]["call_id"] == "call_empty"
assert shell_calls[0]["action"]["commands"] == []
shell_outputs = [entry for entry in replay if entry["type"] == "shell_call_output"]
expected_output: dict[str, Any] = {"type": "shell_call_output", "call_id": "call_empty", "output": []}
if max_output_length is not None:
expected_output["max_output_length"] = max_output_length
assert shell_outputs == [expected_output]


@pytest.mark.parametrize("streaming", [False, True])
def test_empty_local_shell_call_parsing_emits_pair_without_execution(streaming: bool) -> None:
client = OpenAIChatClient(model="test-model", api_key="test-key")
executed_commands: list[str] = []

def local_exec(command: str) -> str:
executed_commands.append(command)
return command

local_tool = OpenAIChatClient.get_shell_tool(func=local_exec)
item = ResponseFunctionShellToolCall.model_validate({
"type": "shell_call",
"id": "sh_empty",
"call_id": "call_empty",
"status": "completed",
"action": {"commands": []},
"environment": {"type": "local"},
})
if streaming:
event = ResponseOutputItemDoneEvent(
type="response.output_item.done", item=item, output_index=0, sequence_number=1
)
update = client._parse_chunk_from_openai(event, options={"tools": [local_tool]}, function_call_ids={})
finalized = ChatResponse.from_updates([update])
assert [content.type for content in finalized.messages[0].contents] == ["shell_tool_call", "shell_tool_result"]
contents = update.contents
else:
response = MagicMock(
output=[item],
output_parsed=None,
metadata={},
usage=None,
id="resp_empty",
model="test-model",
created_at=1000000000,
status="completed",
incomplete=None,
)
parsed = client._parse_response_from_openai(response, options={"tools": [local_tool]})
contents = parsed.messages[0].contents

assert [content.type for content in contents] == ["shell_tool_call", "shell_tool_result"]
assert [content.call_id for content in contents] == ["call_empty", "call_empty"]
assert contents[1].outputs == []
assert executed_commands == []


@pytest.mark.parametrize(
("environment", "tool_name", "item_id", "call_id", "commands"),
[
({"type": "container_reference"}, "run_shell", "sh_empty", "call_empty", []),
({"type": "local"}, None, "sh_empty", "call_empty", []),
(None, None, "sh_empty", "call_empty", []),
({"type": "local"}, "run_shell", None, "call_empty", []),
({"type": "local"}, "run_shell", "sh_empty", None, []),
({"type": "local"}, "run_shell", "sh_empty", "call_empty", None),
({"type": "local"}, "run_shell", "sh_empty", "call_empty", [1]),
],
)
def test_empty_shell_result_is_not_synthesized_for_hosted_unconfigured_or_malformed_calls(
environment: dict[str, str] | None,
tool_name: str | None,
item_id: str | None,
call_id: str | None,
commands: Any,
) -> None:
client = OpenAIChatClient(model="test-model", api_key="test-key")
item = SimpleNamespace(
type="shell_call",
id=item_id,
call_id=call_id,
environment=environment,
action=SimpleNamespace(commands=commands),
status="completed",
)

contents = client._shell_item_to_contents(item, tool_name)

assert [content.type for content in contents] == ["shell_tool_call"]


@pytest.mark.asyncio
async def test_local_shell_tool_requires_approval_before_function_loop_execution() -> None:
"""An explicit local shell call executes only after approval."""
Expand Down
Loading