pathwise

Trajectory evaluation for AI agents. Grade the path, not just the destination.

Archived, August 2026

This library is no longer maintained. What it does is now covered better elsewhere, and it is more useful to say so than to leave the repo looking alive.

What it was. A pytest plugin for asserting an agent's trajectory rather than its output: that one tool was called before another, that a tool was never called, that a run stayed under a step budget, that it did not loop, and that errors were surfaced rather than silently recovered from. Assertions ran across N repeated runs and reported a pass-rate distribution instead of a single boolean, because agents are stochastic and one run tells you very little.

What changed. A survey of the field on 2026-08-23 found each of those capabilities shipped, most within the preceding four months:

• DeepEval — AgentLoopDetectionMetric (May 2026), fully deterministic: tool-call repetition, reasoning stagnation, and DFS cycle detection over the call graph. ToolPermissionMetric (July 2026) for allow/deny lists. Python, pytest-native, Apache-2.0, shipping weekly.
• Braintrust — agentAssertionScorer (2026-07-06): toolOrder, notCalledTool, usedNoTools, maxToolCalls. TypeScript only.
• promptfoo — trajectory:tool-sequence and trajectory:step-count, with a not- prefix for negation. Requires OpenTelemetry wiring.
• Anthropic — claude plugin eval in the Claude Code CLI ships tool_order, tool_used with min/max, max_turns and runs. Experimental and undocumented at the time of writing.
• EvalView and Inspect AI — repeated-run reporting well past what this did: pass@k, pass^k, flakiness scores, confidence intervals, score reducers.

One narrow gap remains: expressing "A must precede B" as a standalone constraint, without supplying a complete reference trajectory. Only promptfoo, Anthropic's CLI, and the unmaintained mcp-eval do that today. It is a real gap. It is also one assertion, which is not a library.

Use instead. DeepEval for Python. promptfoo for Node, or if you already run OpenTelemetry.

What held up. The design bet was that trajectory assertions should be deterministic invariants rather than a reference path graded by an LLM judge — "this must hold whatever route the agent takes," not "the agent should have done these seven things in this order." Reference-path matching is brittle and penalises valid alternative routes, and the field moved against it during 2026. Several teams converged on a near-identical assertion vocabulary over the following months, which suggests the problem was real even though this is not the implementation that will matter.

The code is correct as of archiving and the tests pass. Left readable rather than deleted.

Most agent eval tools grade the final output. Agents don't fail there — they fail on the way. (Written early 2026; by August the field had largely closed this gap — see the archive note above.)

from pathwise_eval import trajectory_eval, render_text

@trajectory_eval(factory=run_refund_agent, runs=20, threshold=0.9)
def refund_flow(traj):
    traj.assert_called("lookup_order").before("issue_refund")
    traj.assert_called("check_refund_eligibility").before("issue_refund")
    traj.assert_never_called("delete_customer")
    traj.assert_steps(max=5)
    traj.assert_no_loops()
    traj.assert_surfaced_errors()

print(render_text(refund_flow()))

refund_flow  FAIL  (20 runs, threshold 90%)

  called_lookup_order                           ████████████████░░░░  16/20  ⚠
      └ not called
  check_refund_eligibility_before_issue_refund  ████████████████░░░░  16/20  ⚠
      └ 'check_refund_eligibility' was never called
  never_called_delete_customer                  ████████████████████  20/20  ✓
  steps_max_5                                   ████████████████████  20/20  ✓
  no_loops                                      █████████████████░░░  17/20  ⚠
      └ lookup_order x2
  surfaced_errors                               ███████████████░░░░░  15/20  ⚠
      └ 1 error(s) hit, output does not acknowledge them

  steps  min=2  p50=4  p95=5  max=5

That agent issues refunds without checking eligibility 20% of the time. Every one of those runs returned a correct-looking answer. A single-run eval would be green.

Install

Not on PyPI yet. Install from source:

pip install git+https://github.com/YashShelar007/pathwise.git

Or for local development:

git clone https://github.com/YashShelar007/pathwise.git
cd pathwise && pip install -e ".[dev]"
pytest

No required dependencies. A tool claiming vendor neutrality shouldn't drag in a provider SDK.

Why trajectories

Agents fail in ways output assertions cannot see:

Failure	What the output looks like
Right answer, wrong reasoning	Correct. It guessed, or read a stale cache. Breaks next week.
Wrong tool	Correct. It called delete_user instead of deactivate_user.
Path bloat	Correct. Twelve steps where three would do; your bill triples.
Loops	Correct. Same call four times, then a plausible summary.
Error masking	Correct. A tool 500'd, the agent recovered silently, your integration is broken.
Drift	Correct — until a model upgrade changes how it solves things.

Two ideas

Structural assertions first. Reading the trace is free, instant, and deterministic. Most real bugs are caught there. LLM-as-judge is reserved for what structure genuinely can't answer — because tools that lead with judge calls become too slow and expensive to run in CI, and an eval nobody runs is worth nothing.

Agents are stochastic, so one run proves nothing. The unit of truth is a rate, not a boolean. runs=20 and "passes 16/20" is what tells you whether a prompt change actually helped. A threshold converts that back to pass/fail for CI.

Assertions

# Tool usage
traj.assert_called("search").before("summarize")
traj.assert_called("search").times(max=2)
traj.assert_never_called("delete_account")
traj.assert_only_called("search", "summarize")

# Shape
traj.assert_steps(max=6, min=2)
traj.assert_no_loops(window=3)

# Errors
traj.assert_no_errors()
traj.assert_surfaced_errors()   # errors hit must be acknowledged in the output

# Budget & output
traj.assert_cost(max_usd=0.05)
traj.assert_output_contains("refund", "confirmation")

# Anything else
traj.assert_that("used_cheap_model", traj.trajectory.metadata["model"] == "haiku")

Assertions record rather than raise, so every assertion is evaluated on every run and you get a complete distribution instead of stopping at the first failure.

pytest

# tests/test_agent.py
from pathwise_eval import trajectory_eval

@trajectory_eval(factory=run_agent, runs=10, threshold=0.9)
def test_agent_checks_before_acting(traj):
    traj.assert_called("verify").before("execute")

pytest tests/

The plugin registers automatically and prints the distribution on failure.

Adapters

Bring your own trace:

from pathwise_eval.adapters.claude_agent_sdk import to_trajectory
traj = to_trajectory(messages)

Or build directly:

from pathwise_eval import Trajectory
traj = Trajectory.from_calls("lookup_order", ("issue_refund", {"id": "A-1"}))

Trajectory is a plain dataclass — adapting any framework is a small function. LangGraph and OpenTelemetry adapters are next.

What this does not do

Being honest about this is more useful than a longer feature list.

• No semantic quality grading yet. Judge-based assertions are designed but not in v0.1. Structural assertions can't tell you whether a summary was good.
• assert_surfaced_errors uses keyword matching. It catches the common case; a genuinely evasive agent can slip past it.
• Loop detection is signature-based. An agent looping with slightly different arguments each time won't be caught.
• No hosted service, no UI. The CLI and the report are the product.
• It won't tell you what to assert. You still have to know what good behavior looks like.

Status

v0.1, alpha. The API will change. Issues and PRs welcome — especially adapters and failure modes I haven't thought of.

License

Apache-2.0