fix(ci): make the docs-bot changelog exemption actually match by eleshar · Pull Request #3683 · lightspeedwp/.github

eleshar · 2026-09-28T17:18:39Z

Linked issues

Refs #3603 and #3448 — both opened by the docs bot, both failing Require changelog or skip label for the same reason.

Closes nothing: this fixes a gate in the repository that governs automation, not a customer-facing defect, and the two pull requests above were unblocked separately with the meta:no-changelog label rather than by this change.

Context

Severity/Impact: Medium. A required check cannot be satisfied by the input it is meant to exempt, so legitimate bot pull requests are blocked from merging.
Affected versions/environments: current develop and any consuming repository.
Scope: one comparison in .github/workflows/changelog-unified.yml, plus a test.

Reproduction

Open any pull request authored by app/lightspeed-docs-bot that touches at least one non-Markdown file, so the docs-only exemption does not apply.

Require changelog or skip label fails:

##[error]PR requires a CHANGELOG.md update or the meta:no-changelog label.

Observed on chore: Update branch validation metrics #3603 and chore(docs): regenerate impacted README files #3448.

The gate logs no exemption message at any point, which is the tell: the branch that should have returned early was never entered.

Root Cause

The gate compared the author against the GraphQL login shape while reading the webhook payload, which carries the REST shape:

$ gh api repos/lightspeedwp/.github/pulls/3603 --jq '.user.login'
lightspeed-docs-bot[bot]

Shape	Form	Source
REST / webhook	`lightspeed-docs-bot[bot]`	`pull_request.user.login` in the event payload
GraphQL	`app/lightspeed-docs-bot`	`author.login`

changelog-unified.yml:74 compared against app/lightspeed-docs-bot, so it never matched. The same file had the mirror-image problem at line 65, listing app/dependabot as an alternative for Dependabot: the Dependabot exemption still worked because the correct form sat beside the dead one, so the bug was invisible there and fatal for the docs bot.

This survived review because gh pr view --json author returns the GraphQL form, so checking the string by hand gives the reassuring answer.

Fix Summary

Both exemptions now compare a normalised slug, so they hold whichever shape GitHub supplies:

const slug = (login) => login.replace(/^app\//, "").replace(/\[bot\]$/, "");
if (slug(author) === "dependabot") { … }
if (slug(author) === "lightspeed-docs-bot") { … }

That also retires the dead app/dependabot alternative, which carried the same ambiguity in the opposite direction.

The two other bot exemptions in the repository are not affected and are left untouched: pr-template-routing.yml:152 and ai-feedback-validation.yml:63 already use the webhook form, which is why those checks passed on the very same pull requests. A test now pins them so a future edit cannot reintroduce the mismatch there.

Verification

tests/js/bot-author-exemptions.test.js extracts the slug arrow function out of the workflow and executes it against both shapes for both bots, rather than pattern-matching the text. It also asserts near-miss slugs and ordinary accounts are not swept in.

npx jest --config .jest.config.cjs tests/js/bot-author-exemptions.test.js — 7 passed

Non-vacuity checked: against develop's current workflow, 4 of 7 fail, including both shape-agreement tests:

● changelog gate bot exemptions › declares a slug normaliser so the exemption is shape-independent
● changelog gate bot exemptions › matches both exempted bots in either shape
● changelog gate bot exemptions › does not exempt a human author
● changelog gate bot exemptions › no longer compares against the GraphQL-only app/ form

actionlint .github/workflows/changelog-unified.yml — clean
npm run validate:workflows — 14 passed, 0 failed
npx jest --config .jest.config.cjs — 297 suites, 6151 tests, 0 failures
semgrep --config p/security-audit --config p/secrets --config p/php on both changed paths — 0 findings
coderabbit review --base develop — 2 files reviewed, no findings

The bare forms dependabot and lightspeed-docs-bot are deliberately matched: the bare slug is what both shapes reduce to, and both names are reserved so a human account cannot hold them.

Risk & Rollback

Risk level: Low. One comparison function and two literals in a single workflow, plus its test. No schema, no state, no consumer API.
Blast radius: the change makes an existing exemption start firing. That is the intended effect, and it is scoped to the two bot slugs. A human author is unaffected, which the test asserts.
Rollback plan: revert. The workflow reads only the event payload, so there is nothing to unwind.

Changelog

Entry added under CHANGELOG.md → [Unreleased] → Fixed:

Docs-Bot Changelog Exemption — The changelog gate now matches the docs bot in either login shape, so its exemption applies instead of being unreachable code. Pull requests opened by the docs bot with no user-facing content no longer need a changelog entry or a skip label. (#3603, #3448)

Scope

Workflow fix, its test, and the changelog entry. Not bundled with #3604 or #3532.

Summary by CodeRabbit

Bug Fixes
- Pull requests from Dependabot and the documentation bot are now recognized consistently across login formats, so bot changes without user-facing content no longer require a changelog entry.
Tests
- Added checks to verify bot exemptions and ensure human and similar-looking accounts are not exempted.

The changelog gate exempted the docs bot by comparing the pull request author against "app/lightspeed-docs-bot", which is the shape GraphQL reports. The webhook payload this gate reads carries the REST shape, "lightspeed-docs-bot[bot]", so the comparison never matched and the exemption was dead code. Every docs-bot pull request fell through to the changelog requirement, including the fixed-branch README regeneration and metrics pull requests, which have no user-facing content to record. Observed on #3603 and #3448, both opened by app/lightspeed-docs-bot, both failing `Require changelog or skip label`. The other two bot exemptions in the repository are unaffected and stay as they are: pr-template-routing.yml:152 and ai-feedback-validation.yml:63 already use the webhook form, which is why those checks passed on the same pull requests. Rather than swapping one literal for another, both exemptions now compare a normalised slug, so they hold whichever shape GitHub supplies. That also retires the dead "app/dependabot" alternative beside the Dependabot check, which had been carrying the same ambiguity in the opposite direction. tests/js/bot-author-exemptions.test.js extracts the normaliser from the workflow and executes it against both forms for both bots, asserts near-miss slugs and ordinary accounts are not swept in, and pins the two already-correct workflows to the webhook form so a future edit cannot reintroduce the mismatch there. Verified non-vacuous: 4 of its 7 tests fail against the previous version of the workflow.

qodo-code-review · 2026-09-28T17:18:45Z

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

coderabbitai · 2026-09-28T17:18:50Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: lightspeedwp/.github/.coderabbit.yml

Review profile: CHILL

Plan: Advanced

Run ID: a504967e-9bd4-4c7e-ab88-4a10463c6dc4

📥 Commits

Reviewing files that changed from the base of the PR and between a2633a8 and fc1f7a3.

📒 Files selected for processing (3)

.github/workflows/changelog-unified.yml
CHANGELOG.md
tests/js/bot-author-exemptions.test.js

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The changelog workflow normalizes pull request author logins before checking Dependabot and docs-bot exemptions. Tests cover both login forms, selected non-exempt logins, and related workflow checks.

Changes

Changelog bot-author exemptions

Layer / File(s)	Summary
Normalize author logins `.github/workflows/changelog-unified.yml`	The gate removes the `app/` prefix and `[bot]` suffix before matching the `dependabot` and `lightspeed-docs-bot` slugs.
Verify exemptions and document the fix `tests/js/bot-author-exemptions.test.js`, `CHANGELOG.md`	Tests check both bot login forms, selected non-exempt logins, gate messages, and login forms in related workflows. The changelog records the docs-bot exemption.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to fc1f7

The bot-login exemption change is ready to merge after normal checks.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to fc1f7

Docs-bot pull requests will now bypass the changelog gate even when their files or labels change. Other review requirements remain, but the scope of this exemption and the result of skipping a required validation job need confirmation.

Retained concerns

Medium · security · observed: The newly effective docs-bot exemption uses PR author alone. It returns before conflicting-label, restricted-change-type, and changed-file checks, and prevents downstream changelog validation for any files subsequently present on that bot-authored PR. Whether human updates to such PRs should retain this exemption is unresolved.

Security review details

Security Blast Radius

inferred — The newly affected scope is PRs recognized as authored by the docs bot against develop or main, not arbitrary human-authored PRs. The develop ruleset’s review requirements remain separate from this changelog exemption.

Security Findings and Attack Paths

inferred — If a bot-authored PR later contains non-documentation or restricted-type changes, its unchanged author identity still takes the early return and bypasses the changelog gate’s content and label controls. The evidence does not establish who can make such updates or that an unauthorized merge results.

Trust Boundaries and Controls

observed — The gate trusts the PR author supplied by GitHub’s event payload, then uses the matching slug as sufficient authority to skip validation. Separate required review and status-check rules exist for develop; the skipped-job status behavior is not established by those rules.

Resilience and Maintainability Implications

observed — The regression test checks login equivalence and near misses in isolation, leaving the exemption’s output propagation and required-check result without an integrated test in the inspected evidence.

Hardening Proposals

proposed — Define which bot-authored changes may receive the exemption, keep label-policy checks outside any author-only early return where they must apply universally, and verify the required-check outcome for an exempt PR.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. (2 skipped: 2…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: fixing the docs-bot changelog exemption so it matches correctly.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. (2 skipped: 2 unsupported.)

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Warning

Some tools did not complete. Review the errors below.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

tests/js/bot-author-exemptions.test.js

(node:2) ESLintIgnoreWarning: The ".eslintignore" file is no longer supported. Switch to using the "ignores" property in "eslint.config.js": https://eslint.org/docs/latest/use/configure/migration-guide#ignore-files
(Use node --trace-warnings ... to show where the warning was created)

Oops! Something went wrong! :(

ESLint: 10.10.0

ReferenceError: module is not defined in ES module scope
This file is being treated as an ES module because it has a '.js' file extension and '/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
at file:///.eslintrc.js?mtime=1790616401133:1:1
at ModuleJob.run (node:internal/modules/esm/module_job:437:25)
at async node:internal/modules/esm/loader:639:26
at async dynamicImportConfig (/node_modules/eslint/lib/config/config-loader.js:185:17)
at async loadConfigFile (/node_modules/eslint/lib/config/config-loader.js:275:9)
at async ConfigLoader.calculateConfigArray (/node_modules/eslint/lib/config/config-loader.js:594:23)
at async #calculateConfigArray (/node_modules/eslint/lib/config/config-loader.js:369:19)
at async Promise.all (index 0)
at async findFiles (/node_modules/eslint/lib/eslint/eslint-helpers.js:637:25)
at async ESLint.lintFiles (/node_modules/eslint/lib/eslint/eslint.js:1025:21)

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-28T17:18:51Z

No description provided.

Required by the release automation for the workflow change in the previous commit, which the changelog gate demands for any non-documentation pull request. Entry under [Unreleased] -> Fixed, referencing #3603 and #3448, the two pull requests the dead exemption blocked.

github-actions · 2026-09-28T17:21:54Z

PR Template Routing

Branch Type: fix
Scope: docsbot-exemption-author-form
Template: pr_bug.md
Labels Applied: type:bug

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-28T17:22:05Z

📋 Changelog Quality Validation

Metric	Count
✅ Passing	145
❌ Failing	10
🆕 New failures in this PR	0
📦 Pre-existing failures	10

Status

✅ Validation PASSED - No new failures introduced by this PR.
Note: 10 pre-existing failure(s) remain in the Unreleased section.

No action required.

The changelog quality gate reported one new critical failure, CHK_MAX_LENGTH: the entry was 403 characters against a 250 limit. Trimmed to 214, keeping the issue references and the user-facing effect. Verified with the gate's own validator rather than by inspection: `.github/validation/changelog` reports Non-Compliant 10, which is the pre-existing baseline on develop, so this entry introduces no new failures.

linear-code · 2026-09-29T04:02:35Z

GIT-2467

eleshar requested review from a team and ashleyshaw as code owners September 28, 2026 17:18

github-actions Bot added priority:normal status:needs-review type:bug lang:js area:testing area:ci area:jest meta:needs-changelog labels Sep 28, 2026

github-actions Bot added the area:documentation label Sep 28, 2026

coderabbitai Bot approved these changes Sep 28, 2026

View reviewed changes

eleshar merged commit 759dbb8 into develop Sep 29, 2026
41 checks passed

eleshar deleted the fix/docsbot-exemption-author-form branch September 29, 2026 04:02

Navigation Menu

fix(ci): make the docs-bot changelog exemption actually match - #3683

fix(ci): make the docs-bot changelog exemption actually match#3683
eleshar merged 3 commits into
developfrom
fix/docsbot-exemption-author-form

eleshar commented Sep 28, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

qodo-code-review Bot commented Sep 28, 2026

Uh oh!

coderabbitai Bot commented Sep 28, 2026 •

edited

Loading

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

linear-code Bot commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

eleshar commented Sep 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked issues

Context

Reproduction

Root Cause

Fix Summary

Verification

Risk & Rollback

Changelog

Scope

Summary by CodeRabbit

Uh oh!

qodo-code-review Bot commented Sep 28, 2026

Uh oh!

coderabbitai Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Security Architecture Review

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 28, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Changelog Quality Validation

Status

Uh oh!

Uh oh!

linear-code Bot commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

eleshar commented Sep 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading