STAC-25584: restore the omnibus git and bazel caches in the DEB build - #491
Open
LouisParkin wants to merge 2 commits into
Open
STAC-25584: restore the omnibus git and bazel caches in the DEB build#491LouisParkin wants to merge 2 commits into
LouisParkin wants to merge 2 commits into
Conversation
Only the Go module cache survived the port from GitLab, re-implemented as the godeps-cache image (STAC-25429). The omnibus git cache and the bazel caches were dropped, so every DEB build recompiles from source: summing the omnibus Builder/NetFetcher timings in a green run, roughly 31 of the 49 minutes is compile/fetch, and bazelisk rebuilds openssl (140s) and zlib (30s) every time. That is paid twice per nightly across two architectures on self-hosted runners. OMNIBUS_GIT_CACHE_DIR was not set at all, so tasks/omnibus.py disabled the git cache entirely. Setting it cannot reach the S3 path that upstream assumes: use_remote_cache additionally requires CI_JOB_NAME_SLUG and S3_OMNIBUS_GIT_CACHE_BUCKET, and neither exists on GitHub, so the local cache is used and the KeyError documented in UPSTREAM_MERGE.md stays unreachable. The --install-directory /opt/stackstate-agent override is what keeps the cache path branded; without it the post-build purge would git -C against a nonexistent /opt/datadog-agent. Cache writes are deliberately bounded rather than a one-to-one port of the GitLab layers. The repository cache is capped at 10 GB with LRU eviction, and the omnibus git cache is the highest value per byte, so: - keys are content-addressed (omnibus definitions, release.json; bazel version and lockfile) with no run id, and save is gated on a non-exact restore, so an unchanged build reuses the entry instead of writing a fresh multi-GB copy; - saves are skipped on pull_request, so PRs read the cache without competing for the cap or paying the upload; - only bazelisk, bazel/install and the repository cache are persisted, not the whole output_user_root. XDG_CACHE_HOME now derives from BAZEL_CACHE_ROOT so the build and the cache steps cannot drift apart. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Setting OMNIBUS_GIT_CACHE_DIR makes omnibus's git cache reachable for the first time on GitHub Actions, and its incremental cache step commits each built software definition into the cache repository. The build container runs as root with no user.name/user.email, so that commit aborted the build with "unable to auto-detect email address". The identity matches the one the repo already uses for bot commits in tasks/libs/common/utils.py. Passing it through GIT_AUTHOR_*/GIT_COMMITTER_* rather than `git config --global` keeps it scoped to the job and avoids mutating the container's global git state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only the Go module cache survived the port from GitLab (as the godeps-cache image).
OMNIBUS_GIT_CACHE_DIRwas not set at all, sotasks/omnibus.pydisabled the git cache entirely, and nothing persisted the bazel caches — so every DEB build recompiles from source. About 31 of the 49 minutes of a green run is omnibus compile/fetch, paid twice per nightly across two architectures on self-hosted runners.Setting
OMNIBUS_GIT_CACHE_DIRcannot reach the S3 path upstream assumes:use_remote_cachealso needsCI_JOB_NAME_SLUGandS3_OMNIBUS_GIT_CACHE_BUCKET, neither of which exists on GitHub, so the local cache is used and theKeyErrorinUPSTREAM_MERGE.mdstays unreachable.Writes are bounded rather than a one-to-one port of the GitLab layers, because the repo cache is capped at 10 GB with LRU eviction: keys are content-addressed with no run id and save is gated on a non-exact restore, saves are skipped on
pull_request, and onlybazelisk,bazel/installand the repository cache are persisted — not the wholeoutput_user_root.Reviewer check: because saves are skipped on PRs, this PR can only show that omnibus stops reporting
OMNIBUS_GIT_CACHE_DIRas missing and that the restore steps are harmless. The speed-up itself is only observable on the second default-branch build after merge.Jira: https://stackstate.atlassian.net/browse/STAC-25584