Skip to content

docs: note on non-transparent wrapper evaluators under GraphEvaluator - #260

Merged
ptomecek merged 1 commit into
mainfrom
docs/wrapper-evaluator-graph-note
Aug 24, 2026
Merged

docs: note on non-transparent wrapper evaluators under GraphEvaluator#260
ptomecek merged 1 commit into
mainfrom
docs/wrapper-evaluator-graph-note

Conversation

@ptomecek

Copy link
Copy Markdown
Collaborator

Summary

Adds a docs subsection to the "Write a custom evaluator" how-to (docs/wiki/how-to/Cache-Results.md) covering a subtle footgun when a non-transparent wrapper evaluator (one that reshapes, replaces, or publishes the result) is combined with GraphEvaluator.

GraphEvaluator re-drives both __deps__ resolution and child-node evaluations through the same evaluator stack. A wrapper that fires on every invocation therefore:

  • runs on the GraphDepList payload of __deps__ (not a model result) — reshaping logic such as .df access crashes there; and
  • side-effects on dependency results it should not own — double-publishing and changing cache identity so it no longer matches the true compute.

What the note says

  • Gate the wrapper on context.fn == "__call__" so it never acts on __deps__ (includes a small PublishingEvaluator example).
  • Scope the override to specific models / model_types to avoid firing on dependencies — but note this is not sufficient on its own, because a model still re-drives its own __deps__, so the fn == "__call__" guard is always required.
  • Points at the built-in DryRunEvaluator as a worked example (passes through on __deps__, uses a re-entry guard for child drives).

The findings were verified empirically against a GraphEvaluator stack.

Notes

  • Docs-only change; no code changes.
  • mdformat --check and codespell pass.

Add a subsection to the "Write a custom evaluator" how-to explaining that a
result-reshaping or publishing wrapper evaluator must gate on
context.fn == "__call__". Because GraphEvaluator re-drives __deps__ and child
nodes through the same evaluator stack, a wrapper that fires on every
invocation crashes on the GraphDepList payload of __deps__ and side-effects
child results (double-publish, cache-identity mismatch).

Documents that scoping the override to specific models/model_types stops
firing on dependencies but is not sufficient on its own, since a model still
re-drives its own __deps__, and points at DryRunEvaluator as a worked example.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.48%. Comparing base (6a788a6) to head (6328113).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #260   +/-   ##
=======================================
  Coverage   93.48%   93.48%           
=======================================
  Files         176      176           
  Lines       20327    20327           
  Branches     1350     1350           
=======================================
  Hits        19002    19002           
  Misses       1052     1052           
  Partials      273      273           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Test Results

    1 files  ±0      1 suites  ±0   3m 8s ⏱️ +8s
1 328 tests ±0  1 326 ✅ ±0  2 💤 ±0  0 ❌ ±0 
1 334 runs  ±0  1 332 ✅ ±0  2 💤 ±0  0 ❌ ±0 

Results for commit 6328113. ± Comparison against base commit 6a788a6.

♻️ This comment has been updated with latest results.

@ptomecek
ptomecek marked this pull request as ready for review August 24, 2026 12:09
@ptomecek
ptomecek merged commit f6ac5d3 into main Aug 24, 2026
20 checks passed
@ptomecek
ptomecek deleted the docs/wrapper-evaluator-graph-note branch August 24, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant