Add manual, diff-scoped live-test trigger for PRs - #3478
Draft
Kostya Linou (linouk23) wants to merge 5 commits into
Draft
Add manual, diff-scoped live-test trigger for PRs#3478Kostya Linou (linouk23) wants to merge 5 commits into
Kostya Linou (linouk23) wants to merge 5 commits into
Conversation
Live tests run only nightly on main today, so breakage caused by a PR is found
post-merge. Running the full suite per PR is too expensive, and the one existing
manual promotion ("Run live integration tests") runs *all* groups.
This adds a second manual Semaphore promotion, "Run relevant live tests (PR)",
that a maintainer can click on a PR build. It detects which live-test groups the
diff touches and runs only those, reusing the existing CLI_LIVE_TEST_GROUPS
build-tag machinery -- so you get a pre-merge signal for the changed commands
without waiting for nightly.
- scripts/live-test-affected-groups.sh maps changed files to groups: shared code
(pkg/, cmd/, go.mod) -> all; test/live/*_live_test.go -> its own build tag;
internal/<dir> -> group via an explicit table (kafka, schema-registry, connect,
flink, iam[+core], login/logout->auth, api-key/environment/organization->core).
CLI live tests are not co-located with command code, so the dir->group table is
the thing to keep in sync as groups are added. Unmapped dirs have no live
coverage and don't trigger a run.
- .semaphore/live-tests-pr.yml runs `make live-test CLI_LIVE_TEST_GROUPS=<detected>`,
8h limit, auto-cancels on new pushes.
Manual by design: no per-commit cost, and prod secrets run only when a trusted
maintainer triggers it. Nightly `all` remains the backstop for cross-group blast
radius and backend/API drift, which pre-merge testing can't catch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
There was a problem hiding this comment.
Pull request overview
Adds a manual Semaphore promotion that can run a diff-scoped subset of Confluent Cloud live-test groups on PR builds, using a new helper script to detect affected groups from the PR’s diff against the base branch.
Changes:
- Introduces
scripts/live-test-affected-groups.shto computeCLI_LIVE_TEST_GROUPSfrom changed paths and//go:buildtags intest/live/*_live_test.go. - Adds a new manual Semaphore pipeline (
.semaphore/live-tests-pr.yml) that detects affected groups and runsmake live-testfor only those groups (or skips when none). - Registers the new manual promotion entry in
.semaphore/semaphore.yml.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| scripts/live-test-affected-groups.sh | New diff-analysis helper to derive CLI_LIVE_TEST_GROUPS (or NONE / all) from changed files. |
| .semaphore/semaphore.yml | Adds a new manual promotion entry pointing to the PR-scoped live tests pipeline. |
| .semaphore/live-tests-pr.yml | New manual pipeline that fetches secrets, detects affected groups, and runs the corresponding live tests. |
Suppressed comments (1)
scripts/live-test-affected-groups.sh:74
- The "no changed files" check only strips literal spaces (
${CHANGED// }), so a whitespace-only value containing newlines/tabs can slip through and be processed as if it had changes. Since CHANGED is newline-delimited, stripping all whitespace is more robust.
if [ -z "${CHANGED// }" ]; then
log "no changed files vs $BASE_REF"; echo "NONE"; exit 0
fi
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Makes the scoped PR run test-level precise, matching the provider PR.
CLI live tests are testify suite methods (func (s *CLILiveTestSuite)
TestKafkaTopicCRUDLive()) run as subtests of TestLive, and they are not
co-located with command code, so the detector now:
- maps each changed internal/<dir> to its live-test group(s);
- narrows, within those groups, to the test/live file(s) whose name shares a
token with the changed command file (e.g. internal/kafka/command_topic* ->
kafka_topic_live_test.go), and asks a new AST helper (scripts/livetestfuncs)
for the exact suite methods they declare;
- runs precisely those via -run '^TestLive$/^(TestKafkaTopicCRUDLive|...)$',
wired through a new CLI_LIVE_TEST_RUN override on `make live-test`.
It stays whole-group when a change can't be narrowed to specific test files
(e.g. internal/kafka/command_broker*, which has no matching live test), and
blast-radius (all) for shared code (pkg/, cmd/, go.mod), so it never under-runs.
Nightly stays the backstop. The detector now prints two lines, GROUPS=<...> and
RUN=<regex|empty>.
CLI_LIVE_TEST_RUN must be exported, not passed as a make argument, because make
strips the trailing `$` anchor from command-line variable values. The group
accumulator is deliberately not named GROUPS (a reserved bash variable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test/dry-run override was expanded unquoted, so an override value was subject to word-splitting and pathname globbing (e.g. 'pkg/*' would expand against the cwd). Quote it and normalize spaces to newlines so the override is taken literally. Only affects the CHANGED_FILES_OVERRIDE path; CI uses the git diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Open
14 tasks
…n layer For a button a maintainer clicks on their own PR, per-test precision wasn't worth the machinery. This replaces the AST helper + *_LIVE_TEST_RUN Makefile override + two-line detector output with a plain name->group table. The detector now prints ONE line (NONE / all / comma-separated groups) and the pipeline runs 'make live-test' with them, reusing the existing group filtering and RTCE handling. One click, no input; anything unrecognized (shared file, new resource) runs everything, so it never skips a test it should run. Removes scripts/livetestfuncs and reverts the Makefile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the need to click the manual button: a new block in semaphore.yml runs the affected live tests automatically on the merge queue's gh-readonly-queue/* branch, so a PR can't merge if its live tests fail. It reuses the same detector. - run/when limits the block to gh-readonly-queue/* branches, so normal PR runs skip it and this pipeline's existing required check passes fast (lets the PR enter the queue); on the queue branch the block runs and gates the merge. - Depends on the main test block, so live tests run only if the build/tests pass. - Pipeline execution_time_limit raised to a 12h ceiling for the queue run; normal runs finish quickly and are unaffected. Requires repo-admin setup (enable merge queue on the base branch; keep this pipeline's check required) and validation that Semaphore builds gh-readonly-queue branches -- otherwise the required check would never report and deadlock the queue. The manual promotion stays as an optional on-demand run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release Notes
No user-facing changes — CI tooling only. The CLI binary is unchanged.
Checklist
This PR only adds CI configuration and a helper script (no command behavior changes), so the command/prod items below are not applicable.
Whatsection below whether this PR applies to Confluent Cloud, Confluent Platform, or both.Test & Reviewsection below.Blast Radiussection below.What
This PR applies to Confluent Cloud CI only.
The problem: live tests only run once a night on
main. So if a PR breaks a command, we don't find out until the next day — after it has already merged. Running every live test on every PR would be too slow and expensive.What this adds: a new button in Semaphore, "Run relevant live tests (PR)", that a maintainer can click on the open PR, before it is merged. Instead of running everything, it looks at what the PR changed and runs only the live tests for the areas the PR touched (e.g. just Kafka, or Kafka + Flink). So you can confirm your change still passes its live tests before merging it — instead of finding out the next morning from nightly, after it's already on
main.How it decides what to run (one click, no input needed):
internal/kafka), it runs that area's live tests. The mapping is a small folder→area table in the script.go.mod,pkg/,cmd/), it safely runs everything.It never runs too little: anything it can't pin down precisely falls back to the whole group, or to all. The nightly full run stays as the safety net. Nothing runs automatically — the pipeline only runs when a maintainer clicks the button, so there is no extra cost per commit.
Files changed (just three):
scripts/live-test-affected-groups.sh— reads the diff and prints which areas to test..semaphore/live-tests-pr.yml— the new (manual) pipeline..semaphore/semaphore.yml— adds the one button.Blast Radius
No customer impact — this is CI-only tooling and changes no command code. The new pipeline is manual (does nothing unless a maintainer clicks it) and additive; the existing nightly and smoke pipelines are untouched apart from one added button. Worst case on a manual run is a slightly wider or narrower set of tests than intended; the nightly run is unaffected.
Test & Review
Not yet run through a live pipeline — this PR is the mechanism. The detection logic was checked against the real repo:
Opened as draft to agree on the approach first. Next step: click the button on a throwaway PR to confirm it works end to end.