Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,31 @@ during the 2.x migration are no longer part of the TinyAgents workspace.
The unreleased entries below describe the changes accumulated before this
major release.

## Unreleased: tool hook denials answer the model

**Breaking (behaviour).** A `tinyagents-hooks` tool-hook `Deny` no longer
aborts the run. `ToolHooksMiddleware` now returns
`TinyAgentsError::ToolFailed`, so the agent loop answers the call with a failed
tool result carrying `tool hook '<hook>' denied <tool>: <reason>` and the run
continues; the tool never runs. This matches `HookPermission::Deny`'s
contract ("The agent is told why") and covers blank or invented tool names a
host denies, which used to end the turn before the unknown-tool policy ran.

Hosts that relied on a deny ending the run must set `continue_: Some(false)`
on the decision. That form stops the run with the new
`TinyAgentsError::ToolRefused`, classified as `TerminalReason::ToolRefused` and,
for hosted callers, `HostedErrorKind::ToolRefused` with the message
`hosted agent invocation stopped: host policy refused a tool call` instead of
the generic `hosted agent invocation failed`. The refusal text never carries
the call's arguments. `Ask` is unchanged and still fails the run with
`TinyAgentsError::Tool`.

An answered deny releases its tool-call slot like every other `before_tool`
refusal, so a model that keeps calling a denied tool is bounded by
`RunLimits::max_model_calls`. `TinyAgentsError`, `TerminalReason` and
`HostedErrorKind` are `#[non_exhaustive]`, so the new variants are additive.
See `docs/modules/hooks/README.md`.

## Unreleased: model observations report who served a call and its first token

**Breaking.** `TurnObservation::Model` and `TurnObservation::ModelDispatch`
Expand Down
14 changes: 14 additions & 0 deletions crates/tinyagents-harness/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,20 @@ pub enum TinyAgentsError {
#[error("permanent tool failure: {0}")]
ToolFailed(String),

/// Host policy refused a tool call and asked for the **run** to stop,
/// rather than for the model to be told and carry on.
///
/// A `before_tool` refusal answered to the model is
/// [`Self::ToolFailed`]; this variant is the fatal form, raised for
/// example by a `tinyagents-hooks` deny carrying `continue: false`. The
/// payload names the refusing hook, the tool and the reason, never the
/// call's arguments. The run ends with
/// [`crate::terminal::TerminalReason::ToolRefused`], and a hosted caller
/// sees [`crate::runtime::HostedErrorKind::ToolRefused`]. Never retried by
/// [`crate::retry::is_retryable`].
#[error("tool call refused: {0}")]
ToolRefused(String),

/// A tool (from [`tinytools::Tool::execute`]) or a `before_tool`
/// middleware asked for **human approval** before this call runs (A2).
///
Expand Down
34 changes: 31 additions & 3 deletions crates/tinyagents-harness/src/runtime/agent.rs
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,10 @@ pub enum HostedErrorKind {
Policy,
/// The model provider failed the call.
Provider,
/// Host tool policy refused a call and stopped the run
/// ([`TinyAgentsError::ToolRefused`]). A refusal answered to the model
/// does not end the run and never surfaces here.
ToolRefused,
/// Any other internal failure not covered by a more specific kind.
Internal,
}
Expand Down Expand Up @@ -167,6 +171,7 @@ fn classify_hosted_error(error: &TinyAgentsError) -> HostedErrorKind {
| TinyAgentsError::PhysicalDispatchAdmission(_) => HostedErrorKind::LimitExceeded,
TinyAgentsError::Validation(_) | TinyAgentsError::Steering(_) => HostedErrorKind::Policy,
TinyAgentsError::Provider(_) | TinyAgentsError::Model(_) => HostedErrorKind::Provider,
TinyAgentsError::ToolRefused(_) => HostedErrorKind::ToolRefused,
_ => HostedErrorKind::Internal,
}
}
Expand All @@ -191,7 +196,25 @@ fn hosted_error_message(kind: HostedErrorKind) -> &'static str {
HostedErrorKind::LimitExceeded => "hosted agent invocation exceeded a configured limit",
HostedErrorKind::Policy => "hosted agent invocation was rejected by policy",
HostedErrorKind::Provider => "hosted agent invocation failed at the model provider",
HostedErrorKind::Internal => "hosted agent invocation failed",
HostedErrorKind::ToolRefused => {
"hosted agent invocation stopped: host policy refused a tool call"
}
HostedErrorKind::Internal => GENERIC_HOSTED_FAILURE,
}
}

const GENERIC_HOSTED_FAILURE: &str = "hosted agent invocation failed";

/// The fixed message for a failed hosted stream, chosen from the run's typed
/// terminal outcome because the stream carries only the raw error text.
fn hosted_stream_failure_message(
outcome: Option<&crate::terminal::TerminalOutcome>,
) -> &'static str {
match outcome.map(|outcome| outcome.reason) {
Some(crate::terminal::TerminalReason::ToolRefused) => {
hosted_error_message(HostedErrorKind::ToolRefused)
}
_ => GENERIC_HOSTED_FAILURE,
}
}

Expand Down Expand Up @@ -243,6 +266,7 @@ impl From<HostedError> for TinyAgentsError {
HostedErrorKind::StreamIdleTimeout => TinyAgentsError::StreamIdleTimeout(error.message),
HostedErrorKind::LimitExceeded => TinyAgentsError::LimitExceeded(error.message),
HostedErrorKind::Policy => TinyAgentsError::Validation(error.message),
HostedErrorKind::ToolRefused => TinyAgentsError::ToolRefused(error.message),
HostedErrorKind::Provider | HostedErrorKind::Internal => {
TinyAgentsError::Model(error.message)
}
Expand All @@ -254,6 +278,10 @@ impl From<HostedError> for TinyAgentsError {
#[path = "agent_dispatch_refusal_tests.rs"]
mod dispatch_refusal_tests;

#[cfg(test)]
#[path = "agent_tool_refusal_tests.rs"]
mod tool_refusal_tests;

/// A host-owned turn request.
///
/// `agent_id` is opaque to the harness. It is resolved only through the host
Expand Down Expand Up @@ -454,7 +482,7 @@ impl<State: Send + Sync + 'static, Ctx: Send + Sync> Stream for AgentStream<'_,
fn sanitize_hosted_stream_item(mut item: AgentStreamItem) -> AgentStreamItem {
match &mut item {
AgentStreamItem::Failed { error, run } => {
*error = "hosted agent invocation failed".to_string();
*error = hosted_stream_failure_message(run.terminal.as_ref()).to_string();
// The partial run carries its typed outcome; keep the
// classification, drop the raw failure text.
if let Some(outcome) = run.terminal.as_mut() {
Expand Down Expand Up @@ -497,7 +525,7 @@ fn sanitize_hosted_event(record: &mut EventRecord) {
*error = "hosted middleware failed".to_string();
}
AgentEvent::RunFailed { error, outcome, .. } => {
*error = "hosted agent invocation failed".to_string();
*error = hosted_stream_failure_message(outcome.as_ref()).to_string();
// The typed outcome mirrors the raw error text; keep its
// classification (reason, class, phase) and drop the detail.
if let Some(outcome) = outcome {
Expand Down
126 changes: 126 additions & 0 deletions crates/tinyagents-harness/src/runtime/agent_tool_refusal_tests.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
use super::{
HostedError, HostedErrorKind, hosted_error, sanitize_hosted_event, sanitize_hosted_stream_item,
};
use crate::agent_loop::AgentStreamItem;
use crate::error::TinyAgentsError;
use crate::events::{AgentEvent, EventRecord};
use crate::ids::{EventId, RunId};
use crate::middleware::AgentRun;
use crate::terminal::{TerminalOutcome, TerminalReason, TimeoutPhase};

const REFUSED: &str = "hosted agent invocation stopped: host policy refused a tool call";

fn refusal() -> TinyAgentsError {
TinyAgentsError::ToolRefused("tool hook 'allow_list' denied secret: forbidden".into())
}

fn refused_run() -> AgentRun {
let mut run = AgentRun::new();
run.terminal = Some(TerminalOutcome::from_error(
&refusal(),
TimeoutPhase::AfterTurn,
));
run
}

#[test]
fn tool_refusal_is_classified_as_its_own_terminal_reason() {
let outcome = TerminalOutcome::from_error(&refusal(), TimeoutPhase::AfterTurn);
assert_eq!(outcome.reason, TerminalReason::ToolRefused);
assert_eq!(outcome.class, crate::terminal::TerminalClass::Failure);
}

#[test]
fn hosted_error_names_a_tool_refusal_without_its_detail() {
let hosted = hosted_error(&refusal(), refused_run());

assert_eq!(hosted.kind, HostedErrorKind::ToolRefused);
assert_eq!(hosted.message, REFUSED);
let outcome = hosted.run.as_ref().and_then(|run| run.terminal.as_ref());
assert_eq!(
outcome.map(|outcome| outcome.message.as_str()),
Some(REFUSED)
);
assert_eq!(
outcome.map(|outcome| outcome.reason),
Some(TerminalReason::ToolRefused)
);
}

#[test]
fn hosted_tool_refusal_round_trips_to_its_own_variant() {
let hosted = HostedError {
kind: HostedErrorKind::ToolRefused,
message: REFUSED.to_owned(),
timeout_bound: None,
dispatch_refusal: None,
run: None,
};
assert!(matches!(
TinyAgentsError::from(hosted),
TinyAgentsError::ToolRefused(message) if message == REFUSED
));
}

#[test]
fn tool_refusal_is_not_retryable() {
assert!(!crate::retry::is_retryable(&refusal()));
}

#[test]
fn hosted_stream_failure_names_a_tool_refusal() {
let item = sanitize_hosted_stream_item(AgentStreamItem::Failed {
error: refusal().to_string(),
run: Box::new(refused_run()),
});
let AgentStreamItem::Failed { error, run } = item else {
panic!("still a failure");
};
assert_eq!(error, REFUSED);
assert_eq!(
run.terminal
.as_ref()
.map(|outcome| outcome.message.as_str()),
Some(REFUSED)
);
}

#[test]
fn hosted_stream_failure_stays_generic_for_other_errors() {
let mut run = AgentRun::new();
run.terminal = Some(TerminalOutcome::from_error(
&TinyAgentsError::Tool("boom".into()),
TimeoutPhase::AfterTurn,
));
let item = sanitize_hosted_stream_item(AgentStreamItem::Failed {
error: "tool error: boom".into(),
run: Box::new(run),
});
assert!(matches!(
item,
AgentStreamItem::Failed { error, .. } if error == "hosted agent invocation failed"
));
}

#[test]
fn hosted_run_failed_event_names_a_tool_refusal() {
let error = refusal();
let mut record = EventRecord {
id: EventId::new("evt-0"),
offset: 0,
event: AgentEvent::RunFailed {
run_id: RunId::new("run"),
error: error.to_string(),
outcome: Some(TerminalOutcome::from_error(&error, TimeoutPhase::AfterTurn)),
},
};
sanitize_hosted_event(&mut record);
let AgentEvent::RunFailed { error, outcome, .. } = &record.event else {
panic!("still RunFailed");
};
assert_eq!(error, REFUSED);
assert_eq!(
outcome.as_ref().map(|outcome| outcome.message.as_str()),
Some(REFUSED)
);
}
10 changes: 8 additions & 2 deletions crates/tinyagents-harness/src/terminal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
//! | `TinyAgentsError::StreamIdleTimeout` (breaker threshold) | `ProviderFailed(Some(StreamIdleTimeout))` | `Timeout` |
//! | provider / model / overflow / empty-response errors | `ProviderFailed(reason)` | `Failure` |
//! | tool errors | `ToolFailed` | `Failure` |
//! | `TinyAgentsError::ToolRefused` (host policy stopped the run) | `ToolRefused` | `Failure` |
//! | run caps, depth caps | `LimitReached(..)` | `Failure` |
//! | steering pause | `Paused` | `Suspended` |
//! | deferred tool calls | `Deferred` | `Suspended` |
Expand All @@ -42,7 +43,7 @@
//! 3. idle / provider-call timeout (`ProviderFailed(Some(Timeout))`)
//! 4. limits (`LimitReached`)
//! 5. no-progress / repeat guard (`Halted`)
//! 6. other failures (`ProviderFailed`, `ToolFailed`, `Internal`)
//! 6. other failures (`ProviderFailed`, `ToolFailed`, `ToolRefused`, `Internal`)
//! 7. suspension (`Paused`, `Deferred`)
//! 8. success (`Completed`)
//!
Expand Down Expand Up @@ -118,6 +119,9 @@ pub enum TerminalReason {
ProviderFailed(Option<FailoverReason>),
/// A tool failed in a way that ended the run.
ToolFailed,
/// Host policy refused a tool call and stopped the run
/// ([`TinyAgentsError::ToolRefused`]).
ToolRefused,
/// Anything the other reasons do not name.
Internal,
}
Expand All @@ -137,6 +141,7 @@ impl TerminalReason {
| Self::Halted
| Self::ProviderFailed(_)
| Self::ToolFailed
| Self::ToolRefused
| Self::Internal => TerminalClass::Failure,
}
}
Expand All @@ -151,7 +156,7 @@ impl TerminalReason {
)) => 2,
Self::LimitReached(_) => 3,
Self::Halted => 4,
Self::ProviderFailed(_) | Self::ToolFailed | Self::Internal => 5,
Self::ProviderFailed(_) | Self::ToolFailed | Self::ToolRefused | Self::Internal => 5,
Self::Paused | Self::Deferred => 6,
Self::Completed => 7,
}
Expand Down Expand Up @@ -285,6 +290,7 @@ impl TerminalOutcome {
E::Tool(_) | E::ToolFailed(_) | E::ToolNotFound(_) | E::ModelRetry(_) => {
Self::new(TerminalReason::ToolFailed, message)
}
E::ToolRefused(_) => Self::new(TerminalReason::ToolRefused, message),
E::LimitExceeded(_)
| E::RecursionLimit(_)
| E::SubAgentDepth(_)
Expand Down
2 changes: 1 addition & 1 deletion crates/tinyagents-hooks/src/command/exec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ fn finish(
Err(error) if definition.fail_closed => {
tracing::warn!("[hooks] {label} failed and is fail-closed; denying: {error}");
HookRun {
output: HookOutput::deny(format!("hook '{label}' could not run: {error}")),
output: HookOutput::deny(format!("hook '{label}' could not run")),
label,
duration,
error: Some(error),
Expand Down
3 changes: 3 additions & 0 deletions crates/tinyagents-hooks/src/command/exec_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@ mod spawner {
assert!(open.error.unwrap().contains("no isolation available"));
let closed = run_with(refuse(), &hook(true)).await;
assert!(closed.output.is_deny());
let reason = closed.output.agent_message.unwrap();
assert!(!reason.contains("no isolation available"), "{reason}");
assert!(closed.error.unwrap().contains("no isolation available"));
}

#[tokio::test]
Expand Down
3 changes: 2 additions & 1 deletion crates/tinyagents-hooks/src/command/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,8 @@ pub struct HookOutput {
/// Text appended to a tool result or to the session's system context.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub additional_context: Option<String>,
/// `false` aborts prompt submission on `beforeSubmitPrompt`.
/// `false` aborts prompt submission on `beforeSubmitPrompt`. On a
/// tool-hook deny, `false` stops the run instead of answering the model.
#[serde(default, rename = "continue", skip_serializing_if = "Option::is_none")]
pub continue_: Option<bool>,
/// Injects another user turn on `stop` / `subagentStop`.
Expand Down
Loading
Loading