feat: private crate registry via JFrog Artifactory (#12) - #684
Merged
Conversation
Eighteen releases in, `publish = false` and no crates.io presence meant
no colleague could depend on this crate. This wires it to Lab271's
Artifactory tenant as a private Cargo registry, opt-in.
Cargo has no per-project registry config that lives outside the working
tree — there is no `jf cargo-config` and no `jf cargo` command at all,
so `.cargo/config.toml` and `CARGO_*` env vars are the only mechanisms
on offer. That matters here because this repository is public and the
Artifactory Cargo index is not anonymously readable:
$ curl -o /dev/null -w '%{http_code}' \
.../api/cargo/lab-cargo-dev/index/config.json
401
The index also advertises `"auth-required": true`, which Cargo honours
for crate downloads as well as index reads. So a committed
`.cargo/config.toml` carrying a source replacement would 401 every
anonymous `cargo build` — every outside contributor, every fork — and
would do it at dependency-fetch time, with an authentication error that
gives the contributor no hint that the fix is to delete a file they did
not add. The npm pilot of the same evaluation rejected a committed
`.npmrc` for exactly this reason.
Hence: commit `.cargo/config.toml.example`, gitignore
`.cargo/config.toml`. The default clone is untouched and resolves from
crates.io as before. ADR-0038 records the decision and the alternatives.
`package.publish` stays `false`, deliberately. It does not need to
change: `cargo package` works under it, and the resulting `.crate`
deploys with `jf rt upload`, which Artifactory indexes into a valid
registry entry on its own (~5s later, with deps/features/cksum parsed
from Cargo.toml). Setting `publish = ["lab-cargo-dev"]` would also work
and would *not* open a path to crates.io — `cargo publish --registry
crates-io` still refuses — but it is unnecessary, and whether this crate
should be publishable at all is a separate decision that is not this
one's to make.
Two things the docs call out because they are not obvious:
- `sqlite-rs` is already taken on crates.io by an unrelated crate, 19
versions up to 0.3.7. `lab-cargo-dev` merges its crates.io cache with
our local repo, so one index path serves 20 versions from two
projects with nothing distinguishing them. Ours resolves correctly
today only because the ranges do not overlap. `lab-cargo-prod` has no
remote and serves 0.18.10 alone.
- Consumers should depend on us via the named-registry form, not the
source replacement. Under source replacement the consumer's lockfile
records our private crate as coming from crates.io, which is untrue
and makes deny.toml's `sources` check pass it silently.
The workflow skips cleanly whenever JF_ACCESS_TOKEN is absent, which is
always the case on forked PRs — a JFrog-caused red build would count
against the evaluation's reliability criterion, whose threshold is zero.
Refs: Lab271/labs-jfrog-poc#12
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`workflow_dispatch` only fires for a workflow that already exists on the default branch, so from a feature branch there is no way to make the publish and promote jobs actually run. An unexecuted workflow is not evidence that it works, and the promote job carries the evaluation's hard-fail assertion (promoted digest identical to the dev digest), so it needs a real run rather than a reading. Reverted in the next commit, before merge. Also makes the consume step skip when the current version is not in lab-cargo-prod yet, which is the normal state on a PR that bumps the version. Failing there would be a JFrog-caused CI failure, and the evaluation's threshold for those is zero. Refs: Lab271/labs-jfrog-poc#12 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Artifactory's Cargo index advertises `"auth-required": true`, and Cargo then refuses to query it at all unless a credential provider is named: "authenticated registries require a credential-provider to be configured". `CARGO_REGISTRIES_*_TOKEN` alone is not enough — the provider has to be listed before the env var is consulted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found the hard way, and it broke a real consumer:
* `cargo package` is not byte-reproducible across machines. Commit
e04330e produced sha256:eb1deeae… on a laptop and sha256:84074b9f…
on a runner.
* `lab-cargo-dev-local` and `lab-cargo-prod-local` are mutable.
Re-uploading 0.18.10 overwrote the bytes silently — no error, no
warning — including in prod, where that version already carried
status `Released`.
* Cargo pins checksums in Cargo.lock, so that is not a harmless
re-upload. A consumer pinned to the old digest now fails outright:
"checksum for `sqlite-rs v0.18.10` changed between lock files ...
unable to verify that `sqlite-rs v0.18.10` is the same as when the
lockfile was generated". Reproduced, not theorised.
Publish and promote now skip cleanly when the version already exists,
rather than clobbering it. Bumping the version is the way to release new
bytes.
Repository immutability on the prod local repo
(Lab271/labs-jfrog-poc#4) is the actual fix and needs platform admin we
do not have. Until then this guard is the only thing between a re-run
and a broken downstream lockfile.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The publish and promote jobs have now run for real on this branch (run 33790369579, all four jobs green, "Promoted bytes identical to dev — no rebuild in the path"), and the overwrite guard has been verified skipping cleanly on a re-run (run 33790771562). The trigger has served its purpose. Back to tags and workflow_dispatch only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
iheitlager
added a commit
that referenced
this pull request
Sep 7, 2026
…supersedes #678) (#693) * feature: spec extension for sqe * spec: autoindex write-side rule (010/Req-8) + incremental row access (013/Req-7) 010 gains Requirement 8: a write MUST NOT succeed while leaving an index it could not read unmaintained. Measured at 0.18.5 against stock sqlite3 3.51.0 -- inserting into a stock-created composite-PK table leaves rows out of sqlite_autoindex_*, after which the oracle undercounts and integrity_check reports rows missing, while the write returns rc=0. Names both acceptable fixes (recover the autoindex column list from the declared constraint, or refuse the write) and scopes autoindex *creation* out to V3/V7. The embedding-api spec gains Requirement 7: statements must yield rows incrementally. It also upgrades the composite-PK prerequisite from inferred to measured, flagging it as the highest-priority item in or around that spec -- it is silent corruption of a valid SQLite file, not an ergonomic gap, and its Req 6 byte-identity scenario cannot pass while it stands. That spec is numbered 012 as of this commit and is renumbered to 013 by the next one, after main landed its own spec 012. Refs: 010/Req-8, 013/Req-7 * docs: renumber embedding-api spec to 013 and its ADR to 0034 Main landed its own spec 012 (`012-query-constraints`) and its own ADR-0033 (constant propagation / OR-to-IN) while this branch was open, so both numbers collided on rebase. The newcomer moves: - `.openspec/specs/012-embedding-api/` -> `013-embedding-api/`, heading updated - `adr/0033-embedding-api-owns-the-connection-driver-out-of-tree.md` -> `0034-`, heading updated, and its five `spec 012` prose references retargeted to 013 - `adr/index.md` gains the 0034 row, which 0ee936b omitted entirely Renumbering ADR-0034 does not violate the immutability convention: it has never been on main and is still `Status: Proposed`. Also fixes four `**Implementation:**` lines in spec 013 that were already marked `(planned)` but written in a form `tools/assurance.py:474` does not match -- its regex only accepts `(planned)` immediately after a *single* backticked path, so `` `a`, `b` (planned) `` scored as active and its not-yet-written test links counted as dead. With that fixed, the dashboard is byte-identical to main: 86 active requirements, Completeness 85/86 (99%), Coverage 270/270 (99%), zero dead links, planned 2 -> 10. Refs: 013/Req-1, 013/Req-7 * docs: renumber the embedding-API ADR to 0041 and reconcile spec 013 with the tree (#678) Takes over #678's spec so it can land off current `main`. Jacob's three commits are cherry-picked verbatim above this one and keep their authorship; everything here is the reconciliation they could not carry, because the tree moved after they were written. ADR renumber, 0034 -> 0041. 0034 was already taken on `main` (`0034-index-range-seeks.md`) when #678 was opened, and 0038/0039/0040 have since gone to the Cargo registry decision (#684), `Value`'s `Arc` payloads (#688) and the streaming primitive (#683). Three citations in spec 013 and the `index.md` row move with it. The index gains a gap at 0039/0040 until those two branches land, which is the correct state for this branch rather than a placeholder. Four claims in spec 013 were true when written and are not now: - **Item 7 said "nothing in `src/vdbe/` offers a step or iterator API".** `vdbe::Execution` is that API as of #683, with `run()` reimplemented as a wrapper over it. Requirement 7 is restated as a facade gap and its `Implementation:` line now points at `Execution::next_row` as the thing to build on -- #682 found the ordering matters, because a facade retrofitted onto `execute_with_db` cannot be made incremental afterwards. - **Item 3 attributed the `!Send` problem to the pager alone.** So did ADR-0041. `Value::Text`/`Blob` held `Rc` payloads, which made a result row -- precisely the thing that has to leave a worker thread -- unsendable too. Closed by #688/ADR-0039, which also narrows Requirement 4: the worker thread is still required for the pager, but it now hands rows across instead of copying them. - **The composite-PK prerequisite was one item; it is two.** #685 fixed maintenance of an existing `sqlite_autoindex_*` and the read-only safety valve (spec 010/Req 8). Creating one on `CREATE TABLE` is still open as #687. SQE's two workarounds split the same way: the write-refusal one comes out now, the dropped-composite-key one waits on #687. - **Requirement 7's acceptance scenario could not pass.** It asked that "peak allocation is proportional to the ten rows". Peak heap for a streaming read is a floor set by the page cache, not a slope in rows pulled, so no correct implementation satisfies that wording. Restated as independence from result size -- 8.68 MB flat against 137.7 MB materialized on 1,000,000 rows (#682) -- with `DEFAULT_PAGE_CACHE_CAPACITY` named as the knob that moves the floor (2000 pages -> 8.68 MB, 64 -> 291 KB, ~4.5% streaming cost). That is both the property a consumer needs and, unlike proportionality, testable. Also corrected in both specs: measurements were cited against "stock `sqlite3` 3.51.0", but the pinned oracle is 3.53.4 (`tests/corpus/oracle.rs:22`, `Cargo.toml [package.metadata.oracle]`). On this machine a bare `sqlite3` is Apple's 3.51.0 codec build, which `tools/gen_fixtures.sh` refuses by design. I re-derived the autoindex rule against the pinned 3.53.4 across eleven DDL shapes, comparing `pragma_index_info` key lists rather than index counts: every case agrees, including the counter-intuitive ones (declaration order beats primary-key-first; a rowid alias consumes no number). The citation was wrong, not the rule -- but "measured against the pinned oracle" is this repo's whole assurance basis, so it has to be accurate. Spec 010/Req 8's three scenario `Tests:` links already name the exact test functions #685 created. They stay `(planned)` here: the flip belongs to the PR that discharges the requirement, and it cannot happen on this branch because the tests do not exist on it. `make check-assurance` passes at 86/86 and 276/276, unchanged -- every requirement added here is `(planned)`, so it is excluded from scoring by design and the dashboard cannot move until the implementing tickets land. No dead links. Refs: 010/Req-8, 013/Req-1, 013/Req-4, 013/Req-7, #678, #682, #683, #685, #687, #688 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: PRAGMA synchronous already has a handler — a fifth stale claim in spec 013 (#678) My reconciliation commit said four of spec 013's claims had stopped being true. There were five. Item 6 read "`Pager` syncs but nothing states what is guaranteed, and `synchronous` has no handler". The second clause was already false when #678 was written: #645 implemented `PRAGMA synchronous` in full — the bare query form reporting `0`/`1`/`2` like stock SQLite, all three levels, and a decided per-level fsync-skip policy recorded in ADR-0036 (`src/vdbe/pragma.rs:79`, dispatched at `src/vdbe/exec.rs:760`, with unit tests). Nothing about it is a stub. That also means Requirement 5 is further along than it claims. It asks the API to "honor `PRAGMA synchronous` at least to distinguish FULL from OFF", which is a weaker ask than what already exists, and says "what is missing is a documented guarantee and any way to trade it" — the trade mechanism is exactly what #645 added. Requirement 5's remaining work is the written guarantee, the transaction surface, and the busy/retryable error handling, not the PRAGMA. I found this while answering "will this work with SQE yet", by checking each of the spec's seven gaps against the tree instead of trusting the list. Worth noting for anyone reviewing #693: the list was written against 0.18.5 and the tree is 0.18.10, so treat every "is missing" line as a claim to re-verify rather than a fact. The four I corrected first were the ones my own branches falsified; this one had been stale for longer and nothing I built touched it. `make check-assurance` unchanged at 86/86 and 276/276, no dead links — Requirement 5 stays `(planned)`, since the requirement as a whole is not discharged even though this part of it is. Refs: 013/Req-5, #645, #678 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jacob Verhoeks <jjverhoeks@schubergphilis.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ilja Heitlager <iheitlager@schubergphilis.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the Cargo half of Lab271/labs-jfrog-poc#12.
Eighteen releases in,
publish = falseand no crates.io presence meant no colleague could depend on this crate. This wires it to Lab271's Artifactory tenant as a private Cargo registry — opt-in, so a default clone is entirely unaffected.The public-repo trade-off, and why the config is not committed
Cargo has no per-project registry config that lives outside the working tree. npm has
.npmrc, and the JFrog CLI can sidestep even that withjf npm-config. There is nojf cargo-configand nojf cargocommand at all —jf --helpsays so outright. So.cargo/config.tomlandCARGO_*env vars are the only mechanisms on offer.That matters because this repository is public and the Artifactory Cargo index is not anonymously readable:
The index also advertises
"auth-required": true, which Cargo honours for crate downloads as well as index reads. A committed source replacement would therefore 401 every anonymouscargo build— every outside contributor, every fork — at dependency-fetch time, with an authentication error that gives the contributor no hint that the fix is to delete a file they did not add.So:
.cargo/config.toml.exampleis committed,.cargo/config.tomlis gitignored. ADR-0038 records the decision and the four alternatives rejected.package.publishstaysfalseIt does not need to change.
cargo packageworks under it, and the resulting.cratedeploys withjf rt upload, which Artifactory indexes into a valid registry entry on its own (~5s later, withdeps,featuresandcksumparsed fromCargo.toml).publish = ["lab-cargo-dev"]would also work, and is narrower than it looks —cargo publish --registry crates-iostill refuses with "The registrycrates-iois not listed in thepackage.publishvalue", and a barecargo publishauto-targets the single allowed registry. But it is unnecessary, and only the upload path produces build-info. Whether this crate should be publishable at all is a separate decision and not this PR's to make.Two things worth knowing before relying on this
sqlite-rsis already taken on crates.io by an unrelated crate — 19 versions, up to0.3.7.lab-cargo-devmerges its crates.io cache with our local repo, so one index path serves 20 versions from two different projects with nothing distinguishing them. Ours resolves correctly today only because the version ranges do not overlap.lab-cargo-prodhas no remote and serves0.18.10alone. Renaming the crate is the real fix, before any public release.Cargo.lockrecords our private crate assource = "registry+https://github.com/rust-lang/crates.io-index"— untrue, and it makesdeny.toml'ssourcescheck (unknown-registry = "deny",allow-registry= crates.io only) pass a private-registry dependency silently.Test plan
Verified against the live tenant, not reasoned about:
cargo fetch --lockedthroughlab-cargo-devCargo.lockunchanged — all 116sourceentries still point at crates.iolab-cargo-dev-local/crates/sqlite-rs/sqlite-rs-0.18.10.crate,sha256:eb1deeae…e04330e8in oneGET /api/build/sqlite-rs/2?project=lab--copy=truelab-cargo-prod-local, digestsha256:eb1deeae…— identical, no rebuildsqlite-rs = "0.18", resolved0.18.10from the registry, compiled and linked it, and ranAnonymous access was confirmed broken before choosing the mechanism (the 401 above), which is the whole basis for ADR-0038.
Notes for a human
jfrog-prodenvironment does not exist yet. GitHub creates it on this workflow's first run with no protection rules, so the promote job is currently ungated. Add required reviewers under Settings → Environments →jfrog-prodbefore treating it as an approval gate. Until then the environment records who triggered the run, not who approved it.JF_ACCESS_TOKENis a repo secret carrying project-admin rights onlab, so any workflow in this repository can promote to prod, and one credential covers both build and promote. That is a least-privilege regression against the OIDC design in labs-jfrog-poc#6 and is recorded there, not fixed here.pushtrigger for this branch —workflow_dispatchonly fires for a workflow that already exists on the default branch, so there was no other way to make the publish and promote jobs actually run. All four jobs went green in run 33790369579 (Promoted bytes identical to dev — no rebuild in the path), and the overwrite guard was verified skipping cleanly in run 33790771562. The trigger is gone from the final state;on:is back to tags andworkflow_dispatch.[registry] global-credential-providers = ["cargo:token"]named explicitly beforeCARGO_REGISTRIES_*_TOKENis consulted at all (not mentioned in Artifactory's Set Me Up), and re-publishing a version silently overwrites released bytes — see the third commit, which is the one worth reading.Xray
Reported in full in the PoC findings, but the short version, since it bears on this repo's existing supply-chain gate:
jf auditdoes not recognise a Rust project. It reports the tree as[unknown]and generates an SBOM with no library components, so the 116-crate closure thatmake check-denyandsqlite-rs-dev.cdx.jsonalready cover is never examined. Xray does have Cargo CVE data and applies it to a.cratescanned withjf scan— confirmed against a deliberately vulnerabletime 0.1.44, which reportsCVE-2020-26235, typecargo— but this crate has zero runtime dependencies, so there is nothing there to find.make check-denyremains the supply-chain gate. Artifactory adds distribution, not assurance. No change todeny.toml, the SBOMs, or any existing gate is proposed here.Spend: matched estimate.
🤖 Generated with Claude Code