Skip to content

[AAASM-6251] 🐛 (ci): Audit the project env, not uvx's own - #351

Open
Chisanan232 wants to merge 1 commit into
mainfrom
v0.0.1/AAASM-6251/fix/audit_project_environment
Open

Chisanan232 wants to merge 1 commit into
mainfrom
v0.0.1/AAASM-6251/fix/audit_project_environment

Conversation

@Chisanan232

Copy link
Copy Markdown
Contributor

What changed

.github/workflows/ci.yaml's dependency-audit job now runs its audit through
uv run --frozen --with pip-audit instead of uvx pip-audit, and allowlists the
single advisory that has no fixed release.

Why

uvx deliberately builds the tool it runs its own throwaway environment, and
pip-audit with no requirements file and no explicit package argument audits the
environment it is itself running in. So uv sync --frozen installed 194 project
packages, and the next step then installed ~29 unrelated packages and audited
those. The gate has been structurally incapable of failing on a project
dependency for its entire lifetime.

It is not a theoretical gap. Job 110761168336 on main at 0d2ac609:

Run uv sync --frozen
Installed 194 packages in 144ms
Run set -euo pipefail
uvx pip-audit \
Installed 29 packages in 18ms
No known vulnerabilities found

That job was green on every commit through 2026-10-02 while this repository
carried 24 open Dependabot alerts against uv.lock, one of them critical.

The job's comment also asserted that "pip-audit resolves the synced
environment". That was false as written, so the comment is corrected too.

How to verify (text only — no visual artifacts)

Differential, back to back on one identical tree. Detached checkout of
7cad098bac5f3abbff95423d94fa0f6fe2542c24, the commit immediately before this
sweep's four security bumps, whose uv.lock carries virtualenv 21.6.1,
pyjwt 2.13.0, urllib3 2.7.0 and gitpython 3.1.59. uv sync --frozen first, then:

command exit result
uvx pip-audit --strict --desc (old) 0 No known vulnerabilities found
uv run --frozen --with pip-audit pip-audit --strict --desc --ignore-vuln PYSEC-2026-3740 (new) 1 Found 24 known vulnerabilities, ignored 1 in 3 packages

Packages flagged by the new command: pyjwt, urllib3, virtualenv. This is
the anti-vacuity proof
: the fixed gate fails on a knowingly vulnerable locked
set that the old gate passed, and the allowlist is narrow enough that only nltk
is suppressed while the other three still bite.

Green on current main for the real reason. Same command against main's
lock: No known vulnerabilities found, 1 ignored, exit 0. Without the allowlist
flag it is Found 1 known vulnerability in 1 package — nltk 3.10.3 PYSEC-2026-3740, reported with an empty fix column.

The allowlisted advisory is genuinely unfixable today. uv tree --frozen --invert --package nltk gives nltk v3.10.3 -> llama-index-core v0.14.23 -> agent-assembly (group: dev), and the PyPI JSON API reports 3.10.3 as the newest
nltk release, so no bump can satisfy it. The flag carries a dated rationale and
an explicit removal condition in the workflow.

Gate wiring is unchanged and still blocking. dependency-audit remains in
ci-success's needs list, and ci-success still fails on a failure result.
No required check was weakened, removed or made non-blocking.

Lint. actionlint .github/workflows/ci.yaml exits 0. The YAML parses and the
gate's run: block, extracted verbatim from the committed file and executed as a
script, reproduces the exit codes above. pre-commit run --files .github/workflows/ci.yaml exits 0 (every hook reports no matching files).

Scope note

The pip-audit finding set is not a superset of Dependabot's. gitpython
3.1.59 was flagged by Dependabot but is not reported by pip-audit on that tree.
Neither surface subsumes the other; both stay necessary.

A second tempting fix, --path .venv/lib/python<X.Y>/site-packages, is
deliberately not used: it is silent on a path that does not exist. Locally
the venv is python3.14 while CI's setup-python pins 3.12, and
uvx pip-audit --path .venv/lib/python3.12/site-packages exited 0 with
No known vulnerabilities found against a lock holding 25 of them. That would
reintroduce exactly this defect the day the interpreter version moved. The
workflow comment records why.

Closes AAASM-6251

🤖 Generated with Claude Code

The gate ran `uvx pip-audit`. `uvx` gives the tool its own throwaway
environment and pip-audit audits whichever environment it runs in, so
the job audited pip-audit's ~29 dependencies and never looked at the 194
packages the preceding `uv sync --frozen` had just installed. It was
therefore incapable of failing on a project dependency, and it did not:
it was green on every commit while this repository carried 24 open
Dependabot alerts against uv.lock, one of them critical.

Run the audit through `uv run --frozen --with` so it resolves the locked
project environment, and allowlist the one advisory that has no fixed
release (nltk PYSEC-2026-3740) in the format the job already documented.

Proven on one identical tree, the pre-bump lock at 7cad098:
`uvx pip-audit --strict --desc` exits 0 with "No known vulnerabilities
found", while the new command exits 1 with "Found 24 known
vulnerabilities, ignored 1 in 3 packages" (pyjwt, urllib3, virtualenv).

refs AAASM-6251

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Oct 2, 2026

Copy link
Copy Markdown

@codecov

codecov Bot commented Oct 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant