Skip to content

Add retrieval example suite for recall@k, MRR and nDCG - #5

Merged
technobasant merged 2 commits into
technobasant:mainfrom
HarshRajSinghania:add-retrieval-example
Oct 16, 2026
Merged

technobasant merged 2 commits into
technobasant:mainfrom
HarshRajSinghania:add-retrieval-example

Conversation

@HarshRajSinghania

Copy link
Copy Markdown
Contributor

Summary

Adds an offline retrieval example that exercises the retrieval scorers end-to-end.

Motivation

Addresses #2: the retrieval scorers were only covered by unit tests. This provides a small labelled corpus and two deterministic retrievers so the scorers, paired comparison and gate can run on a retrieval change.

Implementation

  • examples/retrieval/cases.jsonl: 20 queries with graded relevance judgements.
  • examples/retrieval/retrievers.py: good_retriever (term-overlap ranking) and noisy_retriever (simulates a regression by demoting the top hit on even-length queries).
  • examples/retrieval/suite.yaml: configures the good retriever and gates on nDCG@5.
  • examples/retrieval/README.md: usage instructions.

The example is fully deterministic and requires no models or network, matching the style of examples/deterministic.

Testing

The files were added via the GitHub API. The suite follows the same structure as existing examples and uses the public scorer types (recall_at_k, mrr, ndcg) already registered in the codebase. No local test run was possible in this environment; the change is additive and does not modify existing code.

Closes #2.

HarshRajSinghania and others added 2 commits October 13, 2026 10:08
Addresses technobasant#2 by providing a small labelled corpus and two deterministic retrievers (good and noisy) so the retrieval scorers, paired comparison and gate can be exercised end-to-end.
Drop the unused random import and shorten the noisy retriever's docstring so
ruff passes. The README now says the regression is deterministic (the top hit
drops to rank 5 on even-length queries), switches retrievers with --system
instead of editing the suite, and states what the run shows: recall@5 is
unchanged, MRR and nDCG@5 fall, 9 of 20 queries regress, p = 0.0039, exit 1.
@technobasant
technobasant self-requested a review as a code owner October 16, 2026 18:08

@technobasant technobasant left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. It does what #2 asked, and it makes a good teaching example: moving the top hit to rank 5 leaves recall@5 untouched while MRR and nDCG@5 drop, which is exactly why the three scorers exist.

I pushed one commit on top before merging:

  • removed the unused import random and wrapped the long docstring, so ruff check passes (CI's lint job would have failed on both);
  • the README now describes the regression as deterministic (the top hit drops to rank 5 on even-length queries), switches retrievers with --system instead of editing the suite, and records what the run shows: 9 of 20 queries regress, none improve, exact McNemar p = 0.0039, gate exit 1.

All eight CI jobs pass. Merging.

@technobasant
technobasant merged commit 1e47868 into technobasant:main Oct 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a retrieval example that exercises recall@k, MRR and nDCG

2 participants