chore(ci): Convert bash to Go for CI

[kalverra]

chore(ci): Convert bash to Go for CI

Intent

Replace ad-hoc bash matrix generation (multiline grep regexes and jq pipelines) in GitHub workflows with typed, testable Go commands under tools/ci matrix. Discovering Go test functions with grep is brittle and untestable locally; generating matrices in Go enables type-safe configuration, unit + CLI-level test coverage, and uniform runs-on label formatting across test suites. This is groundwork for DX-5101 (runner strategy control in merge queue and releases), which needs a single parameterized place to resolve runs-on labels and spot flags.

Big Changes

Centralized test matrix generators (ci matrix)

Added ci matrix system, ci matrix in-memory, ci matrix ccip, and ci matrix mixed-env subcommands to the tools/ci CLI, backed by a new internal/matrix package. Each command resolves run ID / run attempt / spot flag from flags, action inputs, or GitHub environment (in that order), validates that a run ID is present (empty run IDs would silently produce invalid runner labels), applies defaults, and emits the matrix as a GitHub Actions output (or stdout JSON with --json for local debugging).

Generating matrices in Go replaces regex-based test discovery with go/parser AST walking (DiscoverGoTestNames): deterministic sorted output, per-test topology/config override tables, multi-entry expansion for tests that run under several topologies, and error handling instead of silent grep misses.

Mixed-env workflow adoption

cre-mixed-env-tests.yaml now calls ci matrix mixed-env instead of a 30-line inline jq script. The hardcoded test list and per-test config overrides moved to internal/matrix/mixedenv.go, preserving the exact previous matrix output. No checkout/setup step is needed in the matrix job: the existing $-style setup-runner-spot action already checks out the repo when missing and installs the ci binary via setup-ci-cli.

Small Changes

• ci matrix ccip --mixed-version-only filters the matrix to mixed-version tests for release rollout runs.
• SanitizeTestID produces legal artifact names from test file names for the in-memory suite.
• Run attempt and spot flag default to 1 and spot=co when unset; run ID is validated as required.
• tools/ci/README.md command table updated with the four new subcommands.
• .github/AGENTS.md reformatted and now points agents at the tools/ci CLI for non-trivial workflow shell logic.

Callouts

• Matrix output parity: the generated matrices must byte-match what the old inline scripts produced — job names, runs-on labels (cpu/ram/family differ per suite), and per-test config overrides all feed directly into runner provisioning and test selection. Golden assertions in internal/matrix tests pin these; reviewers should sanity-check the label formats against the workflows they replace.
• Dead code by design: ci matrix system, in-memory, and ccip are implemented and tested but not yet wired into their workflows — cre-system-tests.yaml, cre-regression-system-tests.yaml, ccip-system-tests.yaml, and integration-in-memory-tests.yml still use inline scripts with TODO(DX-5101) markers. Adoption lands in follow-ups; only mixed-env is live here.
• AST discovery ignores build constraints: _test.go files excluded by //go:build tags still contribute test names (same behavior as the old grep). Known limitation, not a regression.
• In-memory timeout parsing only understands Xm durations; other formats silently default to a 15m job timeout (matches old jq behavior).

[github-actions]

✅ No conflicts with other open PRs targeting develop

[trunk-io]

   

Failed Test	Failure Summary	Logs
Test_CRE_V2_EVM_LogTrigger_Invalid_Address_Regression/EVM.LogTrigger_-_EOA_address_(not_a_contract)_fails_with_another_EOA	The test failed because an EOA address that is not a contract did not behave as expected, causing the test to fail.	Logs ↗︎
Test_CRE_V2_EVM_LogTrigger_Invalid_Address_Regression	The test failed without providing specific error details, indicating an unspecified failure in the test case.	Logs ↗︎

View Full Report ↗︎ ⋅ Docs

[erikburt]

See previous comment.

[Copilot]

🟡 Changes recommended

There are functional issues in the new matrix generators/CLI (missing CRE smoke overrides, CCIP matrix incompatibilities, and Cobra defaults preventing Action inputs) that could break parity or future workflow migrations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Risk Rating: MEDIUM — This changes CI matrix generation logic and wires a workflow to a new CLI codepath; failures here can block or misconfigure CI runs.

This PR migrates several GitHub Actions test-matrix generators from ad-hoc bash/jq into the tools/ci Go CLI, aiming for more reliable discovery and easier test coverage.

Changes:

• Added ci matrix subcommands (system, in-memory, ccip, mixed-env) plus supporting matrix builders (AST-based Go test discovery and structured entries).
• Updated CRE mixed-env workflow to call ci matrix mixed-env instead of inline bash/jq.
• Refreshed tools/ci documentation and .github/AGENTS.md guidance to prefer the Go CLI over complex workflow shell.

Scrupulous human review recommended (high-impact areas):

• Matrix parity vs existing workflow behavior before swapping TODO’d workflows (especially CRE smoke per-test overrides and CCIP matrix shape/fields).
• CCIP test-name semantics (go test -run regex expectations) to ensure matrices actually execute the intended tests.

File summaries

File	Description
tools/ci/README.md	Documents new ci matrix subcommands replacing inline matrix scripts
tools/ci/internal/matrix/system.go	Builds CRE smoke/regression matrices via Go test discovery and per-test overrides
tools/ci/internal/matrix/system_test.go	Unit tests for discovery + CRE smoke/regression matrix generation
tools/ci/internal/matrix/mixedenv.go	Builds mixed-env CRE matrix (hardcoded tests + per-test config overrides)
tools/ci/internal/matrix/mixedenv_test.go	Tests for mixed-env matrix generation
tools/ci/internal/matrix/inmemory.go	Parses .github/in-memory-tests.json into a resolved in-memory test matrix
tools/ci/internal/matrix/inmemory_test.go	Tests for in-memory matrix generation + test ID sanitization
tools/ci/internal/matrix/discover.go	AST-based discovery of Test*/Example* function names in _test.go files
tools/ci/internal/matrix/ccip.go	Builds CCIP system test matrix entries
tools/ci/internal/matrix/ccip_test.go	Tests for CCIP system matrix generation
tools/ci/cmd/root.go	Registers the new matrix command group
tools/ci/cmd/matrix.go	Implements CLI subcommands, resolves common inputs/env, writes GHA outputs
tools/ci/cmd/matrix_test.go	CLI-level tests validating JSON output for each matrix subcommand
.github/workflows/cre-mixed-env-tests.yaml	Replaces inline jq matrix creation with ci matrix mixed-env
.github/AGENTS.md	Updates workflow authoring rules/docs to prefer the Go CI CLI

Review details

Suppressed comments (2)

tools/ci/cmd/matrix.go:195

• file is given a non-empty Cobra default (".github/in-memory-tests.json"), so if file == "" { file = act.GetInput("file") } never runs. This prevents overriding the config path via GitHub Action inputs.

	cmd.Flags().StringVar(&file, "file", ".github/in-memory-tests.json", "Path to in-memory tests configuration JSON")

tools/ci/internal/matrix/ccip.go:59

• TestDeleteCCIPJobs-TestRevokeJobs won't match any Go test name when used as a go test -run regex (it becomes ^(TestDeleteCCIPJobs-TestRevokeJobs)$ in .github/workflows/ccip-system-tests.yaml). The actual tests are TestDeleteCCIPJobs and TestRevokeJobs, so this entry effectively runs nothing. Consider making this field a regex alternation so both tests execute in the same job.

			TestName:        "TestDeleteCCIPJobs-TestRevokeJobs",

• Files reviewed: 15/15 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube