Repository navigation
Corpus: add a real-repo detection case (a pinned, known-vulnerable public repo) #13
Description
Extend the real-repo benchmark (umbra_eval/detection/real_repos.py) with another
public, deliberately-vulnerable app so umbra-eval realrepo covers more ground.
Good candidates: WebGoat (Java), Damn Vulnerable GraphQL App, RailsGoat (Ruby),
Vulnerable Node apps — pick one you can pin to a commit.
Where: add a RealRepoCase(...) with a pinned commit (reproducibility),
languages, and a cited provenance. Conservative expect_at_least is optional
(report-only), not a strict assertion.
Acceptance criteria
-
umbra-eval realrepoclones + scans it and reports findings by category. -
pytest -q+ruffgreen (the real-repo run itself is network-gated / not
asserted in unit tests).
See CONTRIBUTING.md.
Activity
feat(corpus): XXE, path-traversal and Go SSRF cases + pinned real rep…
bkd-dotcom commentedon Aug 18, 2026
Done in #31 (merged).
Two pinned real-repo cases: OWASP WebGoat (Java,
7517acca…) and OWASP RailsGoat (Ruby,0222f7da…). First JVM/Ruby targets here — every existing case is Python or JavaScript.Adding them surfaced that
commit=did nothing.scan_real_repoclones with--depth 1and then rangit checkout <sha>withcheck=False; on a shallow clone the object is absent, so the checkout failed withfatal: unable to read tree, the failure was swallowed, and the scan silently ran against the default-branch tip. A case documented as "pinned for reproducibility" was not pinned.Verified both behaviours against a real shallow clone; it now fetches the object first, and if pinning genuinely cannot be honoured the result note says so rather than passing an unpinned scan off as pinned.
Corpus is now 60 cases at recall 1.0 with 0 false positives, verified against the published core v0.7.0 tag in a clean environment.