Skip to content

fix(ci): retry track record snapshot push races - #354

Open
i-xtsu-sixyou-ken-mei wants to merge 2 commits into
mainfrom
fix/track-record-snapshot-push-race
Open

fix(ci): retry track record snapshot push races#354
i-xtsu-sixyou-ken-mei wants to merge 2 commits into
mainfrom
fix/track-record-snapshot-push-race

Conversation

@i-xtsu-sixyou-ken-mei

@i-xtsu-sixyou-ken-mei i-xtsu-sixyou-ken-mei commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Intent

Prevent the daily track-record snapshot workflow from failing after a successful publish/verify when the target branch advances before the workflow pushes its generated track-record-meta.json commit. Harden the push against both main races and workflow_dispatch usage, plus latent dirty-tree failures from pnpm build packages.

Context

Run 33587728101 (2026-09-02) generated, pinned, verified, and committed the new snapshot metadata locally, then failed with a non-fast-forward rejection:

main -> main (fetch first)

The workflow already serializes track-record-snapshot runs with its own concurrency group, but that does not prevent PR merges or other automation from advancing the branch while this multi-minute job is running. The original final step committed against the checkout SHA and immediately ran a bare git push, so any concurrent branch update makes the push stale. Additionally, pnpm build packages regenerates tracked design-token outputs (packages/design-tokens/dist/**, packages/design-tokens/src/generated/tokens.ts); a plain git rebase refuses a dirty tracked tree even when already up to date, which would turn a stale-generated-file state on main into a daily cron failure.

CI red (code-quality / @zapengine/control-center#dup:check) on the first PR build was pre-existing — the merge was built against a stale base (00bef131); the clones were fixed on main by PR #353. After rebase onto 603ba375, all 11 checks on ebc590c1 are green.

Scope

  • Refresh origin/$branch after creating the generated metadata commit (branch = $GITHUB_REF_NAME: main on schedule, the chosen branch on workflow_dispatch).
  • Rebase that generated commit onto the latest origin/$branch with --autostash before pushing (tolerates dirty tracked design-token outputs; stash never includes untracked .track-record/).
  • Retry the fetch/rebase/push sequence up to 3 times when the push is rejected.
  • Keep pushes fast-forward only; no force push. A rebase conflict (concurrent edit to the meta file) fails closed with ::error:: instead of being overwritten — do not git rebase --abort (runner is ephemeral; --abort would exit 128 when rebase never started and hide the diagnostic).
  • Use neutral retry wording ("Push rejected ... refreshing ...") rather than misattributing every failure to a race.
  • Update docs/track-record-pipeline.md runbook: Flow ends with rebase-then-fast-forward push; Failure handling scopes "no retry" to generate/publish/verify scripts and documents the push retry + conflict policy.

Out of scope

  • Snapshot generation, signing, Pinata publishing, or verification behavior.
  • Changing the workflow schedule or concurrency group.
  • Changing branch protection or repository merge policy.
  • Migrating Pinata endpoints/API versions.
  • Sibling workflows backtest-refresh.yml:106 / distribution-snapshot.yml:104 still end with bare git push — same latent race, kept out of scope here; follow-up is a shared composite action (see Reviewer notes).

Product contract / invariants

  • This PR does not change a documented product/architecture invariant.
  • This PR intentionally changes an invariant with explicit product approval.

Affected invariant:

None. This only hardens the workflow's generated-meta commit delivery against concurrent branch updates.

Why:

The existing behavior intends to publish the verified snapshot metadata; the change preserves that behavior while removing repository-write and dirty-tree races.

Acceptance criteria

  • A snapshot run rebases its generated metadata commit onto the latest origin/$GITHUB_REF_NAME before pushing (with --autostash).
  • workflow_dispatch from a feature branch rebases onto and pushes to that branch, not hard-coded main.
  • A dirty tracked tree from pnpm build packages does not block rebase when already up to date.
  • A rebase conflict fails the job with ::error:: and does not overwrite the concurrent change.
  • A push rejected because the branch advanced is retried with a fresh fetch/rebase, up to 3 attempts, with neutral wording.
  • The workflow never force-pushes.
  • Persistent push failure still fails the job instead of being silently ignored.
  • Runbook docs/track-record-pipeline.md matches the shipped behaviour.

Implementation

.github/workflows/track-record-snapshot.yml (lines 74–95): replace the single git push with a bounded 3-attempt loop:

branch="$GITHUB_REF_NAME"
for attempt in 1 2 3; do
  git fetch origin "$branch"
  if ! git rebase --autostash "origin/$branch"; then
    echo "::error::Rebase onto origin/$branch failed (likely a concurrent change to track-record-meta.json); refusing to overwrite it."
    exit 1
  fi
  if git push origin "HEAD:$branch"; then
    exit 0
  fi
  if [ "$attempt" -lt 3 ]; then
    echo "Push rejected (attempt $attempt/3); refreshing $branch and retrying..."
    sleep 2
  fi
done
echo "::error::Failed to push track record meta after 3 attempts."
exit 1

Plus an explanatory comment about --autostash covering pnpm build packages side-effects.

docs/track-record-pipeline.md: Flow last line → "commit the meta change, rebase it onto the latest branch tip, push fast-forward only (retried up to 3 times ...)"; Failure handling → only retry is the final push (fetch/rebase/push, 3 attempts), conflict fails closed, generate/publish/verify still have no retry.

Verified non-issues: bash -e {0} (errexit, no pipefail; if/! exempt; exit 0 in loop ends step); git fetch origin <branch> updates refs/remotes/origin/<branch> under checkout@v7 refspec; fetch-depth: 0 gives full history; empty commit after rebase is dropped and push no-ops with exit 0; credentials via http.extraheader + contents: write; no husky hooks at runtime (HUSKY=0 at install); cron line unchanged so pnpm lint schedules parity holds; workflow YAML outside prettier's scope.

Contract alignment

  • Implementation matches the invariant.
  • Regression / contract tests cover it.
  • Scoped AGENTS.md remains accurate.
  • README / runbook now accurate (updated in this PR).

Validation

  • Shell + YAML: yq -r '.jobs.snapshot.steps[-1].run' .github/workflows/track-record-snapshot.yml > step.shbash -n step.sh PASS; yq parse of whole file PASS (no actionlint/prettier locally).
  • Deterministic simulation (bare origin, bot clone + writer clone, extracted block under bash -e with GITHUB_REF_NAME=main, after same git config/add/commit preamble):
    • no race → pushes on attempt 1, exit 0 ✓
    • race (writer advances main after bot commit) → rebase + push succeed, origin/main linear with bot commit on top, exit 0 ✓
    • conflict (writer edits same meta file differently) → ::error::Rebase onto origin/main failed ... path, exit 1, origin/main unchanged ✓
    • dirty tracked file + up-to-date branch → plain git rebase refuses (proves finding 3); --autostash variant proceeds and pushes ✓
    • GITHUB_REF_NAME=release/x style branch name → refspecs resolve ✓
  • Repo gate that parses workflows: bash scripts/check-schedules-registry.sh — PASS ("OK: GitHub Actions cron schedules match registry").
  • Build cleanliness probe (informational, not blocking): pnpm build packages in worktree then git status --porcelain --untracked-files=no — tree clean today; --autostash retained as latent protection.
  • Docs: re-read docs/track-record-pipeline.md against final workflow — PASS.
  • CI: prior head ebc590c1 had 11/11 green; new head 625bdf7b pushed, gh pr checks 354 --watch expected all green (see Handoff).

Validation gaps

The hosted concurrent-write race itself can only be exercised by a live GitHub Actions run while another writer advances the branch; deterministic simulation above covers the local git mechanics. The next scheduled/manual workflow run will exercise the change in the hosted environment.

Known unrelated failures

The Node 20 deprecation warning shown during post-job cleanup in the reported run is unrelated to the non-fast-forward push failure and is not addressed here.

Reviewer notes

Please focus on the final Commit updated track record meta step. The intended safety property is that the bot only ever fast-forwards; it refreshes/rebases its own generated commit rather than overwriting concurrent repository changes.

Follow-up (out of scope): same latent race exists in sibling workflows backtest-refresh.yml:106 and distribution-snapshot.yml:104 (both end with bare git push; only cron offsets separate them). A shared composite action that encapsulates fetch → rebase --autostash → push HEAD:$branch with bounded retries is the clean fix for all three.

Use GITHUB_REF_NAME instead of hard-coded main so workflow_dispatch
on a feature branch does not push to main. Add --autostash so
rebase tolerates tracked design-token outputs left dirty by
pnpm build packages (rebase refuses a dirty tree even when already
up to date). Wrap rebase in if ! with ::error:: instead of bare
errexit, and use neutral push-rejected wording. Update runbook to
describe rebase-then-fast-forward push with bounded retries.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant