Skip to content

ci: give the testnet image its own ccache on master - #208

Merged
On1x merged 1 commit into
masterfrom
ci/docker-prod-ccache
Oct 30, 2026
Merged

On1x merged 1 commit into
masterfrom
ci/docker-prod-ccache

Conversation

@chiliec

@chiliec chiliec commented Oct 30, 2026

Copy link
Copy Markdown
Member

On master, docker-main.yml builds the testnet image and then the production image. Both jobs used the same cache key, viz-cache-mount-<os>-main-<hash of both Dockerfiles>-<sha>:

  1. build_testnet runs first, restores the previous commit's entry, builds with -DBUILD_TESTNET=TRUE and saves under this commit's key.
  2. build_prod (needs: build_testnet) gets an exact hit on that key: testnet objects. Every TU whose flags differ is a miss, and since an exact hit skips the save, production's own objects are never stored.

The last four master runs show it: testnet compiles in ~140 s, production in ~920 s, every time.

It also explains why the viz-cache-mount-<os>-main- fallback that #202 added to PR builds didn't speed them up: the newest main- entry is always the testnet one.

Change

  • testnet: own prefix viz-cache-mount-testnet-<os>-main-, keyed on Dockerfile-testnet only;
  • production: keyed on Dockerfile-production only, so the main- prefix that PR builds fall back to only ever holds production objects;
  • ccache -z before the build in both Dockerfiles. The counters live in the cache directory and were restored with it, so ccache -s mixed every past run. That is where the 84–87% "hit rate" in the logs came from.

What to expect: the first master run after merge builds production with a cache from the old keys (testnet objects), then saves its own. From the second run on, production should compile at roughly testnet's speed, and PR Docker builds start from a production cache. docker-pr-build.yml isn't changed. CI on this PR only exercises docker-pr-build.yml (the Dockerfile ccache -z line), so the docker-main.yml effect can only be measured after merge.

docker-main.yml builds the testnet image, then production, and both
used the same cache key (viz-cache-mount-<os>-main-<hash>-<sha>). The
testnet job runs first and saves its BUILD_TESTNET objects under this
commit's key; production then gets an exact hit on them, misses every
TU whose flags differ, and never saves its own cache (an exact hit
skips the save). Last four master runs: testnet compiles in ~140 s,
production in ~920 s.

The PR build's last restore key, viz-cache-mount-<os>-main- (#202), hit
the same testnet cache, which is why it didn't speed PR builds up.

- testnet: own prefix viz-cache-mount-testnet-<os>-main-, keyed on
  Dockerfile-testnet only;
- production: keyed on Dockerfile-production only, so the main- prefix
  that PR builds fall back to only ever holds production objects;
- ccache -z before the build in both Dockerfiles: the counters were
  carried in the restored cache, so "ccache -s" mixed every past run.
@chiliec

chiliec commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

A note on the hit-rate numbers quoted in #206, because they are the case this PR fixes.

ccache stores its statistics counters inside the cache directory. The BuildKit cache mount restores that directory from Actions Cache, so ccache -s after a build reports the restored history plus this run, not this run alone.

The "brand-new PR 2195/2589 = 84.8%" figure is almost exactly what master's production build prints (2196/2589, previous run 2196/2589 as well). That is the restored counter, not that PR's own hits, so the real PR hit rate from the main- fallback is unknown so far.

What the logs do show is wall time. On master, testnet compiles in ~140 s and production in ~920 s on every run. That fits production restoring testnet's objects through the shared key, as described above.

With this PR, ccache -z runs before the build, so from the first run after merge the printed numbers are this build's own. That gives a direct check of whether the production key split works.

@On1x On1x left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the keys: the testnet and production cache-mount entries no longer share a prefix, and the production key is hashed on its own Dockerfile only, so the main- fallback that PR builds restore from can now only hold production objects. The ccache -z stops the restored counters from mixing runs — that is where the inflated 84–87% hit rate came from. CI on this PR exercises docker-pr-build (including the Dockerfile line); the docker-main effect is only measurable after merge, and the stated expectation (production ~140 s from the second run) is a checkable prediction.

@On1x
On1x merged commit 99d2fb6 into master Oct 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants