From 25171dfda0d52f8e1a112041598a92b0aaae0820 Mon Sep 17 00:00:00 2001 From: Jan Kadlec Date: Tue, 1 Sep 2026 09:30:25 +0200 Subject: [PATCH] ci: drive the whole release from a single tag push Releasing needed three manual actions: dispatch bump-version, dispatch netlify-deploy, then hand-tag master. Now bump-version pushes vX.Y.Z at the end of the bump job, and that one tag event triggers build-release and netlify-deploy in parallel. The tag is pushed from the bump job itself rather than the commented-out trigger-release job, which would have tagged the pre-bump commit: its fresh checkout resolves to master as of dispatch time. The push must also carry TOKEN_GITHUB_YENKINS_ADMIN, already used for the master push, because GitHub does not trigger workflows from GITHUB_TOKEN pushes -- the likely reason that job was left disabled. Release branches are now rel/X.Y.Z for every bump type. The old patch/X.Y.Z naming was the repository's only reference to patch/, and it hid patch releases from both the pre-merge pipeline and the docs build, which key off rel/** and rel/* respectively. Adds release-tag-checks, which answers the two questions the release workflows ask about a tag. They are not the same question and they disagree exactly on the backport cases. is_latest, the highest stable vX.Y.Z tag, decides the "Latest" badge, so a patch of an older line does not take it from the current release. is_on_master, an ancestry check against the default branch, decides the documentation deploy: releases are tagged on master, while a patch is branched from a release branch and never merged back. Gating the docs on is_latest instead would have let a patch of the newest line through -- it produces the highest tag but still a tree behind master, and the hugo action checks out the triggering tag, so the deploy would have reverted every documentation change merged since that release. The three workflows are serialized. bump-version is keyed by branch so a hotfix bump stays independent of one from master; netlify-deploy takes one group for its shared deploy target; build-release is serialized because is_latest is computed once per run, so two releases in flight could let the earlier one finish last and take the badge back. None cancel in progress -- an interrupted release is a half-made one. MAINTENANCE.md replaces the three-step release with the single dispatch and documents the previously unwritten procedure for patching an already released version, including why bump-version must not be used for it and that a tag runs the workflows as they exist at that tag, so older release lines need the workflow cherry-picked before tagging. Also quotes $GITHUB_OUTPUT in the bump step, clearing the file's last shellcheck warning, and notes on the draft netlify-deploy-v2 that it must bring the gate along when it takes over. --- .../actions/release-tag-checks/action.yaml | 64 ++++++++++ .github/workflows/build-release.yaml | 32 ++++- .github/workflows/bump-version.yaml | 58 ++++----- .github/workflows/netlify-deploy-v2.yaml | 3 + .github/workflows/netlify-deploy.yaml | 33 ++++++ MAINTENANCE.md | 112 ++++++++++++++++-- 6 files changed, 261 insertions(+), 41 deletions(-) create mode 100644 .github/actions/release-tag-checks/action.yaml diff --git a/.github/actions/release-tag-checks/action.yaml b/.github/actions/release-tag-checks/action.yaml new file mode 100644 index 000000000..92450a4c0 --- /dev/null +++ b/.github/actions/release-tag-checks/action.yaml @@ -0,0 +1,64 @@ +# (C) 2026 GoodData Corporation +name: Release tag checks +description: > + Answers the two questions the release workflows ask about the tag that triggered them. + They are not the same question, and they disagree exactly on the backport cases: + + is_latest -- is this the highest version released so far? Decides the "Latest" badge + on the GitHub release. A patch of an older line (v1.60.1 while v1.73.0 + exists) must not take it. + is_on_master -- is this tag an ancestor of the default branch? Decides whether the + documentation is rebuilt. A patch is branched from a release branch and + never merged back, so its tree is behind master; deploying it with --prod + would revert any documentation merged since that release. + + Requires the repository to be checked out with fetch-depth: 0, so that every tag and the + default branch are present. A shallower checkout fails the ancestry check loudly rather + than answering either question wrongly. On a non-tag ref (e.g. a manual workflow_dispatch) + both outputs are 'true'. + +outputs: + is_latest: + description: "'true' when the triggering tag is the highest v*.*.* tag, otherwise 'false'" + value: ${{ steps.check.outputs.is_latest }} + is_on_master: + description: "'true' when the triggering tag is an ancestor of the default branch, otherwise 'false'" + value: ${{ steps.check.outputs.is_on_master }} + +runs: + using: composite + steps: + - id: check + shell: bash + env: + TAG: ${{ github.ref_name }} + REF_TYPE: ${{ github.ref_type }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + run: | + set -euo pipefail + + if [ "$REF_TYPE" != "tag" ]; then + echo "Ref '$TAG' is not a tag; nothing to guard against." + echo "is_latest=true" >> "$GITHUB_OUTPUT" + echo "is_on_master=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + + # Only stable vX.Y.Z tags count. The trigger glob v*.*.* would also match something + # like v0.0.1-test, which sort -V could pick as the highest -- marking every real + # release from then on as not-latest. A non-stable TAG never equals a stable + # $highest, so it correctly comes out as not-latest without a separate check. + highest=$(git tag -l 'v*.*.*' | { grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' || true; } | sort -V | tail -n 1) + if [ -n "$highest" ] && [ "$TAG" = "$highest" ]; then is_latest=true; else is_latest=false; fi + + if git merge-base --is-ancestor "$TAG" "origin/$DEFAULT_BRANCH"; then + is_on_master=true + else + is_on_master=false + fi + + echo "tag=$TAG highest=$highest is_latest=$is_latest is_on_master=$is_on_master" + { + echo "is_latest=$is_latest" + echo "is_on_master=$is_on_master" + } >> "$GITHUB_OUTPUT" diff --git a/.github/workflows/build-release.yaml b/.github/workflows/build-release.yaml index 22ac4dc72..03399d85b 100644 --- a/.github/workflows/build-release.yaml +++ b/.github/workflows/build-release.yaml @@ -14,6 +14,14 @@ on: tags: - v*.*.* +# One release at a time. Each tag is unique, so runs never collide on the tag itself, but +# is_latest is computed once per run: with two releases in flight, the earlier tag can +# finish last and take the "Latest" badge back from the newer one. Serializing keeps the +# badge in release order. Never cancels -- a cancelled run leaves components half-published. +concurrency: + group: build-release + cancel-in-progress: false + env: COMPONENTS: '["gooddata-api-client","gooddata-pandas","gooddata-fdw","gooddata-sdk","gooddata-dbt","gooddata-flight-server","gooddata-flexconnect","gooddata-pipelines","gooddata-eval"]' @@ -56,10 +64,28 @@ jobs: path: | ${{ matrix.component == 'gooddata-api-client' && format('{0}/dist/', matrix.component) || format('packages/{0}/dist/', matrix.component) }} if-no-files-found: error + + tag-checks: + name: Check the triggering tag + runs-on: ubuntu-latest + permissions: + contents: read + outputs: + is_latest: ${{ steps.check.outputs.is_latest }} + steps: + - name: Checkout + uses: actions/checkout@v5 + with: + fetch-depth: 0 # the checks need every tag and the default branch + - id: check + uses: ./.github/actions/release-tag-checks + github_release: name: Create GitHub release runs-on: ubuntu-latest - needs: build + needs: + - build + - tag-checks permissions: contents: write steps: @@ -83,7 +109,9 @@ jobs: token: "${{ secrets.GITHUB_TOKEN }}" draft: false prerelease: false - make_latest: true + # False for a patch of an older line, so v1.60.1 does not take the badge from + # v1.73.0. Only in force for tags whose tree contains this file -- see MAINTENANCE.md. + make_latest: ${{ needs.tag-checks.outputs.is_latest }} files: | dist/**/*.whl dist/**/*.tar.gz diff --git a/.github/workflows/bump-version.yaml b/.github/workflows/bump-version.yaml index 4d708527d..3e0e20f11 100644 --- a/.github/workflows/bump-version.yaml +++ b/.github/workflows/bump-version.yaml @@ -14,6 +14,14 @@ on: - minor - patch +# One bump per branch at a time, so two dispatches cannot race to bump, merge and tag. +# Keyed by branch rather than globally, so a bump from a hotfix branch is not +# blocked by one from master. Never cancels: interrupting this +# between the master push and the tag push would leave a release half-made. +concurrency: + group: bump-${{ github.ref_name }} + cancel-in-progress: false + permissions: contents: write pull-requests: write @@ -21,8 +29,6 @@ permissions: jobs: bump-version: runs-on: ubuntu-latest - outputs: - new_version: ${{ steps.bump.outputs.new_version }} steps: - name: Checkout uses: actions/checkout@v5 @@ -40,7 +46,7 @@ jobs: id: bump run: | NEW_VERSION=$(uv run python ./scripts/bump_version.py ${{ github.event.inputs.bump_type }}) - echo "new_version=$NEW_VERSION" >> $GITHUB_OUTPUT + echo "new_version=$NEW_VERSION" >> "$GITHUB_OUTPUT" - name: Bump version in documentation run: | @@ -50,40 +56,28 @@ jobs: run: | make release-ci VERSION=${{ steps.bump.outputs.new_version }} - - name: Specify release branch - id: branch - run: | - if [ "${{ github.event.inputs.bump_type }}" == "patch" ]; then - RELEASE_BRANCH="patch/${{ steps.bump.outputs.new_version }}" - else - RELEASE_BRANCH="rel/${{ steps.bump.outputs.new_version }}" - fi - echo "release_branch=$RELEASE_BRANCH" >> $GITHUB_OUTPUT - - name: Create and push the new version ${{steps.bump.outputs.new_version}} + env: + VERSION: ${{ steps.bump.outputs.new_version }} run: | git config user.name github-actions git config user.email github-actions@github.com - git checkout -b ${{ steps.branch.outputs.release_branch }} + + # Every release branch is rel/X.Y.Z, patches included. The docs build + # (scripts/generate.sh) and the pre-merge pipeline both key off rel/**. + git checkout -b "rel/$VERSION" git add -A - git commit -m "Release ${{steps.bump.outputs.new_version}}" - git push origin ${{ steps.branch.outputs.release_branch }} + git commit -m "Release $VERSION" + + # Order matters: the docs build enumerates remote rel/* branches, so + # rel/$VERSION has to be on the remote before the tag starts anything. + git push origin "rel/$VERSION" git checkout master - git merge ${{ steps.branch.outputs.release_branch }} + git merge "rel/$VERSION" git push origin master -# TODO: this part waits for docs build and publish optimization it takes too long (~15 minutes) -# trigger-release: -# needs: -# - bump-version -# - create-release-branch -# runs-on: ubuntu-latest -# steps: -# - name: Checkout -# uses: actions/checkout@v5 -# - name: Push new tag – v${{ needs.bump-version.outputs.new_version }} -# run: | -# git config user.name GitHub Actions -# git config user.email github-actions@github.com -# git tag v${{ needs.bump-version.outputs.new_version }} -# git push origin v${{ needs.bump-version.outputs.new_version }} + # The tag push is the single trigger for build-release and netlify-deploy. + # It works only because the checkout above uses TOKEN_GITHUB_YENKINS_ADMIN -- + # GitHub does not trigger workflows from pushes made with GITHUB_TOKEN. + git tag "v$VERSION" + git push origin "v$VERSION" diff --git a/.github/workflows/netlify-deploy-v2.yaml b/.github/workflows/netlify-deploy-v2.yaml index 07ebe55c7..d29c2a4ea 100644 --- a/.github/workflows/netlify-deploy-v2.yaml +++ b/.github/workflows/netlify-deploy-v2.yaml @@ -1,4 +1,7 @@ name: Netlify Deploy V2 (Draft) + +# TODO: when this replaces netlify-deploy.yaml, bring the tag-checks gate with it +# (see .github/actions/release-tag-checks). on: workflow_dispatch: diff --git a/.github/workflows/netlify-deploy.yaml b/.github/workflows/netlify-deploy.yaml index e516085bb..4ada217a9 100644 --- a/.github/workflows/netlify-deploy.yaml +++ b/.github/workflows/netlify-deploy.yaml @@ -1,9 +1,42 @@ name: Netlify Deploy on: workflow_dispatch: + # Released together with the packages: the tag pushed by bump-version triggers + # this workflow and build-release.yaml at the same time, so docs and packages + # build in parallel. + push: + tags: + - v*.*.* + +# One production deploy at a time, whatever triggered it -- the deploy target is a single +# shared resource, so overlapping runs race to decide what is live. Never cancels: a +# cancelled `netlify deploy --prod` can leave the site partly updated. +concurrency: + group: netlify-prod + cancel-in-progress: false jobs: + tag-checks: + name: Check the triggering tag + runs-on: ubuntu-latest + permissions: + contents: read + outputs: + is_on_master: ${{ steps.check.outputs.is_on_master }} + steps: + - name: Checkout + uses: actions/checkout@v5 + with: + fetch-depth: 0 # the checks need every tag and the default branch + - id: check + uses: ./.github/actions/release-tag-checks + netlify-deploy: + # Only tags that are on master publish documentation: the hugo action checks out the + # triggering tag, so deploying from a patch tag would put an outdated site live. + # See .github/actions/release-tag-checks for why this is not the is_latest check. + needs: tag-checks + if: needs.tag-checks.outputs.is_on_master == 'true' runs-on: ubuntu-latest steps: - name: Checkout diff --git a/MAINTENANCE.md b/MAINTENANCE.md index 71d98e48a..8e25abf50 100644 --- a/MAINTENANCE.md +++ b/MAINTENANCE.md @@ -1,14 +1,112 @@ # Repository maintenance and release ## How to release -* manually run [Bump version & trigger release](.github/workflows/bump-version.yaml) workflow -* after the previous workflow finishes, dispatch the GitHub workflow [Netlify Deploy](.github/workflows/netlify-deploy.yaml) on the `master` branch (takes ~15 minutes) - * The styling of the documentation is taken from the `master` branch. For more details see [generate.sh](scripts/generate.sh). -* after the previous workflow finishes, push tag - * the version should be the same as the one in [Bump version & trigger release](.github/workflows/bump-version.yaml) workflow log - * checkout latest master branch and tag it `vX.Y.Z` - * push the tag to the gooddata/gooddata-python-sdk repository (e.g. `git push vX.Y.Z`) +Manually run the [Bump version & trigger release](.github/workflows/bump-version.yaml) workflow and pick the +bump type. That is the whole release. +The workflow bumps the version, creates the `rel/X.Y.Z` branch, merges it to `master`, and pushes the tag +`vX.Y.Z`. That tag push triggers two workflows in parallel: + +* [Build Python Package and Create Release](.github/workflows/build-release.yaml) — builds every component, + creates the GitHub release, publishes to PyPI, and posts to `#releases`. +* [Netlify Deploy](.github/workflows/netlify-deploy.yaml) — builds and publishes the documentation + (takes ~15 minutes, so the packages reach PyPI well before the docs go live). + +The styling of the documentation is taken from the `master` branch. For more details see +[generate.sh](scripts/generate.sh). + +### Recovering a stuck release +Both downstream workflows key off the tag, so a release that stalled can be resumed by hand: + +* if the tag was never pushed, check out the `Release X.Y.Z` commit on `master`, tag it `vX.Y.Z`, and push the + tag to the gooddata/gooddata-python-sdk repository (e.g. `git push vX.Y.Z`) +* if only the documentation failed, dispatch [Netlify Deploy](.github/workflows/netlify-deploy.yaml) manually; + it does not need the tag + +The tag has to be pushed with a personal access token. GitHub does not trigger workflows from pushes made with +the default `GITHUB_TOKEN`, so a tag pushed by a workflow using it would silently start nothing. + +## How to patch an already released version +Use this whenever a release must contain a specific fix and *not* everything currently on `master` — whether +that is an old line (1.60 while `master` is at 1.73) or the newest one. + +Do **not** use the [Bump version & trigger release](.github/workflows/bump-version.yaml) workflow for this. Its +last step is `git checkout master && git merge`, which would drag the old code and version numbers onto +`master`. Its `patch` bump type means "release master as a patch", not "patch the released line". + +Only the tagging is automated; the rest is manual by nature. + +**Prerequisite:** the fix is already merged to `master`. The patch branch is never merged back, so this is what +keeps the fix from being lost in the next release. + +1. **Pick the base and the new version.** List what the line already has with + `git branch -rl '/rel/1.60.*'`. The base is the newest of them, and the new version increments the + patch component **of that base** — so `rel/1.60.0` gives `1.60.1`, but if the line was already patched to + `rel/1.60.2` the next one is `1.60.3`. The steps below use `1.60.1`; substitute your version throughout. + +2. **Create the release branch first**, so the fix has somewhere to be reviewed into: + ```bash + git fetch + # Branch from the base chosen in step 1, not blindly from X.Y.0 -- on an already-patched + # line that would be rel/1.60.2, and starting from 1.60.0 would drop the earlier fixes. + git checkout -b rel/1.60.1 /rel/1.60.0 + git push rel/1.60.1 + ``` + +3. **Cherry-pick the fix through a pull request:** + ```bash + git checkout -b fix/backport-1.60 rel/1.60.1 + git cherry-pick + git push fix/backport-1.60 + ``` + Open the PR against `rel/1.60.1`. The [pre-merge pipeline](.github/workflows/pre-merge.yaml) runs because it + triggers on `rel/**`. Merge once it is green. + +4. **Bump the version on the release branch.** These commands mirror the *Install dependencies* through + *Bump version in codebase* steps of [bump-version.yaml](.github/workflows/bump-version.yaml) — if that + workflow gains or reorders a step, update this block with it: + ```bash + git checkout rel/1.60.1 && git pull + uv sync --only-group release --locked + uv run python ./scripts/bump_doc_dependencies.py 1.60.1 + make release-ci VERSION=1.60.1 + git add -A && git commit -m "Release 1.60.1" + git push rel/1.60.1 + ``` + `git add -A` rather than `commit -am`, matching the workflow, so a newly created file is not dropped. On an + older line `uv sync --locked` can fail if the lock file predates the current uv; re-lock if so. + +5. **Tag it.** This is the only trigger; everything after it is automatic: + ```bash + git tag v1.60.1 + git push v1.60.1 + ``` + +The release is then built and published exactly like any other. Two things differ, and +[release-tag-checks](.github/actions/release-tag-checks/action.yaml) handles both: the GitHub release does not +take the "Latest" badge from the newest version, and the documentation is not rebuilt — the docs build checks +out the triggering tag, so publishing from one would put an outdated site live. + +> **A tag runs the workflows as they exist *at that tag*, not on master.** Release lines branched before the +> release automation was added therefore run their own older copies, in which `make_latest` is hardcoded to +> `true`. Before tagging such a line, cherry-pick `.github/workflows/build-release.yaml` and +> `.github/actions/release-tag-checks/` onto `rel/X.Y.Z` — otherwise the patch takes the "Latest" badge, which +> also changes what `GET /releases/latest` returns. If you only notice afterwards, untick "Set as the latest +> release" on the GitHub release by hand. Those older copies have no tag trigger on the docs workflow, so the +> documentation is safe either way. + +### What the documentation will show +The docs site keeps the four newest release branches, sorted by `major.minor`, and a section is named after the +`major.minor` only. Consequences worth knowing before someone goes looking: + +* A patch never publishes its own documentation — the deploy is gated on the tag being on master. `rel/1.72.1` + does take over the `1.72` section from `rel/1.72.0`, but only at the next deploy from master: the following + release, or a manual dispatch of [Netlify Deploy](.github/workflows/netlify-deploy.yaml) if you need it + sooner. +* Both branches still occupy a slot of the four, so one patch inside the window drops the site from four + displayed versions to three. +* Patching an old line (`rel/1.60.1` while `master` is at 1.73) falls outside the window entirely and never + appears in the docs. ### How-to dev release To publish current master as a dev release version, use [Dev release from master](.github/workflows/dev-release.yaml) GitHub workflow.