Skip to content

Add Duel performance regression guards - #780

Open
sferik wants to merge 2 commits into
ruby-oauth:mainfrom
sferik:duel
Open

sferik wants to merge 2 commits into
ruby-oauth:mainfrom
sferik:duel

Conversation

@sferik

@sferik sferik commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in duel/ against main and the branch side by side, and fails the build if the branch is slower or allocates more.

The 20 benchmarks follow what an app does with oauth2: building authorize URLs, the token endpoint (auth code, OIDC, client credentials, refresh, JWT assertion and revoke), making API requests with a bearer or query token and following a redirect, building and serializing an AccessToken, parsing JSON, form and XML responses, building errors, and the time it takes to require "oauth2". Every request goes to Faraday's test adapter, so nothing waits on a socket.

To check that it actually catches things, I added one extra allocation to Response#content_type, and Duel failed the 12 benchmarks that parse a body, while revoke and the redirect stayed the same. Some extra work in AccessToken#initialize failed the seven benchmarks that build tokens, at 1.26 to 1.52 times the CPU time. Comparing main with itself passed three times in a row.

Everything besides the benchmarks comes from the kettle-jem template, which now has Duel behind workflows.duel: true (structuredmerge/structuredmerge-ruby#81). This PR turns that on in .structuredmerge/kettle-jem.yml and adds what the template generates: the Perf workflow, .duel.rb, gemfiles/modular/duel.gemfile (loaded by the root Gemfile and a new perf appraisal), the rake duel task, the RuboCop include for .duel.rb, the README badge, and a CHANGELOG entry. Each file matches the template's output, so there are no freeze blocks or per-file overrides, and a full template pass on top of this branch leaves all of it as it is. The template's other updates, which have nothing to do with Duel, are left for the next re-template of main. This is ready to merge once kettle-jem is released with the Duel template.

Duel requires Ruby 3.4, so duel.gemfile only adds it on 3.4 and newer, and older Rubies get a rake duel that only warns. The perf appraisal gives the workflow its bundle, so none of the per-Ruby jobs ever see it. .duel.rb runs each version on the Ruby its own version file names, and when the baseline names a different Ruby, the workflow installs that Ruby with its own bundle too, for every kind of run.

There's also a second commit that changes my name from Erik Michaels-Ober to Erik Berlin in the license, the README, the gemspec's authors and the generated docs. Here's the story.

Copilot AI lite review requested due to automatic review settings October 25, 2026 23:40
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 40245853478

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage remained the same at 97.194%

Details

  • Coverage remained the same as the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 659
Covered Lines: 649
Line Coverage: 98.48%
Relevant Branches: 232
Covered Branches: 217
Branch Coverage: 93.53%
Branches in Coverage %: Yes
Coverage Strength: 68.77 hits per line

💛 - Coveralls

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The workflow has a critical pull-request permission risk, and Duel configuration can skip or fail comparisons under supported change and Ruby-version scenarios.

2 open issues
What changed in this PR

Adds Duel-based performance regression guards for OAuth2, including 20 benchmarks integrated into local development and GitHub Actions. It also updates author and copyright references from Erik Michaels-Ober to Erik Berlin.

Changes:

  • Adds Duel and StackProf dependencies for Ruby 3.4+.
  • Adds benchmarks for authorization, token flows, requests, responses, access tokens, errors, and boot time.
  • Integrates Duel with Rake, Appraisal, RuboCop, and CI reporting.
  • Updates author and copyright metadata in project and generated documentation.
File Description
README.md Adds the Duel badge and updates the copyright holder.
Rakefile Registers the Duel Rake task when available.
oauth2.gemspec Updates the gem author metadata.
LICENSE.md Updates the copyright notice.
gemfiles/​perf.gemfile Defines the performance appraisal bundle.
gemfiles/​modular/​duel.gemfile Adds Duel and StackProf for Ruby 3.4+.
Gemfile.lock Locks Duel, StackProf, dependencies, and the current gem version.
Gemfile Loads the Duel development dependencies.
duel/​token_duel.rb Benchmarks token exchange, refresh, assertion, and revoke flows.
duel/​response_duel.rb Benchmarks JSON, form, XML, and error response handling.
duel/​request_duel.rb Benchmarks authenticated requests and redirects.
duel/​duel_helper.rb Provides shared fixtures, Faraday stubs, and benchmark helpers.
duel/​boot_duel.rb Benchmarks requiring oauth2.
duel/​authorize_duel.rb Benchmarks authorization URL construction.
duel/​access_token_duel.rb Benchmarks access-token construction and serialization.
docs/​index.html Updates generated copyright documentation.
docs/​file.README.html Updates generated README documentation.
docs/​file.LICENSE.html Updates generated license documentation.
Appraisals Adds the performance appraisal definition.
.rubocop.yml Includes the Duel configuration in RuboCop checks.
.github/​workflows/​perf.yml Runs Duel comparisons and posts pull-request reports.
.duel.rb Configures Duel to use each version’s Ruby configuration.

Review effort: Lite

Comment thread .github/workflows/perf.yml Outdated
Comment thread .github/workflows/perf.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved CI and configuration issues can prevent intended comparisons from running reliably and can broaden Duel dependencies beyond the performance appraisal.

3 open issues

Review effort: Lite

Comment thread README.md Outdated
# 🔐 OAuth2

[![Version][👽versioni]][👽version] [![Ruby Users Forum][✉️ruby-forum-top-img]][✉️ruby-forum] [![GitHub tag (latest SemVer)][⛳️tag-img]][⛳️tag] [![License: MIT][📄license-img]][📄license] [![Total downloads][👽dl-ranki]][👽dl-rank] [![CodeCov Test Coverage][🏀codecovi]][🏀codecov] [![Coveralls Test Coverage][🏀coveralls-img]][🏀coveralls] [![QLTY Test Coverage][🏀qlty-covi]][🏀qlty-cov] [![QLTY Maintainability][🏀qlty-mnti]][🏀qlty-mnt] [![CI Heads][🚎3-hd-wfi]][🚎3-hd-wf] [![CI Runtime Dependencies @ HEAD][🚎12-crh-wfi]][🚎12-crh-wf] [![CI Current][🚎11-c-wfi]][🚎11-c-wf] [![CI Truffle Ruby][🚎9-t-wfi]][🚎9-t-wf] [![CI JRuby][🚎10-j-wfi]][🚎10-j-wf] [![Deps Locked][🚎13-🔒️-wfi]][🚎13-🔒️-wf] [![Deps Unlocked][🚎14-🔓️-wfi]][🚎14-🔓️-wf] [![CI Test Coverage][🚎2-cov-wfi]][🚎2-cov-wf] [![CI Style][🚎5-st-wfi]][🚎5-st-wf] [![Apache SkyWalking Eyes License Compatibility Check][🚎15-🪪-wfi]][🚎15-🪪-wf]
[![Version][👽versioni]][👽version] [![Ruby Users Forum][✉️ruby-forum-top-img]][✉️ruby-forum] [![GitHub tag (latest SemVer)][⛳️tag-img]][⛳️tag] [![License: MIT][📄license-img]][📄license] [![Total downloads][👽dl-ranki]][👽dl-rank] [![CodeCov Test Coverage][🏀codecovi]][🏀codecov] [![Coveralls Test Coverage][🏀coveralls-img]][🏀coveralls] [![QLTY Test Coverage][🏀qlty-covi]][🏀qlty-cov] [![QLTY Maintainability][🏀qlty-mnti]][🏀qlty-mnt] [![CI Heads][🚎3-hd-wfi]][🚎3-hd-wf] [![CI Runtime Dependencies @ HEAD][🚎12-crh-wfi]][🚎12-crh-wf] [![CI Current][🚎11-c-wfi]][🚎11-c-wf] [![CI Truffle Ruby][🚎9-t-wfi]][🚎9-t-wf] [![CI JRuby][🚎10-j-wfi]][🚎10-j-wf] [![Deps Locked][🚎13-🔒️-wfi]][🚎13-🔒️-wf] [![Deps Unlocked][🚎14-🔓️-wfi]][🚎14-🔓️-wf] [![CI Test Coverage][🚎2-cov-wfi]][🚎2-cov-wf] [![CI Style][🚎5-st-wfi]][🚎5-st-wf] [![Apache SkyWalking Eyes License Compatibility Check][🚎15-🪪-wfi]][🚎15-🪪-wf] [![Duel](https://raw.githubusercontent.com/sferik/duel-ruby/main/assets/badge.svg)](https://github.com/sferik/duel-ruby)
Copilot AI lite review requested due to automatic review settings October 26, 2026 00:38
@pboling

pboling commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

@sferik So far I've never had a case of templated workflows needing to diverge from the template, so I've kept those changes all in the template, and all templated projects (a hundred or more now) then inherit the benefits / hardening.

Duel sounds awesome, and it sounds like it would be a great fit in the kettle-jem template.
So the changes can be added here, and as long as they are added to the template the same way, then re-templating will work fine.

The issue is they are set to "accept_template" - and this choice is due to the rapid pace of development of the template (there have been hundreds of releases) but only intermittent retemplating of each project, and it just made more sense to ensure the template inherited all changes in a normalized manner, than to fight "merging" issues (even though the point of the template is to make merge issues as simple and manageable as possible - it is still friction).

If you have interest in adding duel to the template, I'm happy to review. Otherwise I can try my hand at it.

https://github.com/structuredmerge/structuredmerge-ruby/tree/main/gems/kettle-jem

Bottom line - I think this is fine, and the template can be adjusted to conform and support these changes (without the need for kettle-freeze tags), before it is retemplated.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved moderate workflow, benchmark-fixture, Rake-loading, and generated-file issues remain.

2 open issues
2 resolved since last review

Review effort: Lite

Comment thread .github/workflows/perf.yml
Copilot AI lite review requested due to automatic review settings October 26, 2026 00:50
@sferik

sferik commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@pboling That makes sense, thanks for explaining. I'd be happy to add Duel to the kettle-jem template. I'll make it opt-in, so nothing changes for projects that don't turn it on. Then I'll update this PR to match what the template generates and drop the kettle-jem.yml override, so re-templating oauth2 just works.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved moderate findings could disable benchmark coverage or prevent baseline comparisons from running correctly.

3 open issues

Review effort: Lite

Comment thread .github/workflows/perf.yml
@pboling

pboling commented Oct 26, 2026

Copy link
Copy Markdown
Member

@sferik I've pushed all my recent changes to kettle-jem

@sferik

sferik commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@pboling Thanks! I rebased onto your latest and opened structuredmerge/structuredmerge-ruby#81, which adds Duel to the kettle-jem template behind workflows.duel: true. Once that's in a release, I'll update this PR to just turn it on and drop the overrides.

pboling pushed a commit to structuredmerge/structuredmerge-ruby that referenced this pull request Oct 26, 2026
Add Duel (https://github.com/sferik/duel-ruby) to the template, off by
default, so that every templated project can opt in with
`workflows.duel: true` and re-templating keeps it. This replaces the
per-project setup reviewed in
ruby-oauth/oauth2#780.

With Duel on, the full template generates:

- .github/workflows/perf.yml: a read-only Duel job and a separate
  Comment job with only pull-requests: write, which runs none of the
  branch's code. The baseline's Ruby, and the bundle for it, are
  installed for every event type when its version file (.ruby-version,
  .tool-versions, or mise.toml, in Duel's order) names another Ruby
  than the candidate's.
- gemfiles/modular/duel.gemfile: Duel and StackProf on Ruby 3.4 or
  later only, loaded by the root Gemfile and a new perf appraisal.
- a `rake duel` task in the Rakefile, with a stub for older Rubies.
- the RuboCop include for .duel.rb, and the Duel README badge.
- .duel.rb, only when it is missing.

While Duel is off, kettle-jem neither writes nor removes .duel.rb,
perf.yml, or duel.gemfile, and generated files are unchanged.
actions/download-artifact joins the workflow pin table, and a
workflow.duel changelog transfer filter scopes the replayed entry to
projects that opt in.
@pboling

pboling commented Oct 26, 2026

Copy link
Copy Markdown
Member

Merged structuredmerge/structuredmerge-ruby#81 - now working on fixing CI there, then release, then I'll retemplate here on main, then you can rebase this PR on main, and some of this work will fall off... unless you want to try templating yourself on this branch.

Duel runs the benchmarks in duel/ on the main branch and on a change
side by side, and fails the Perf workflow when the change is slower.
The 20 benchmarks cover building authorize URLs, the token endpoint
(auth code, OIDC, client credentials, refresh, JWT assertion, revoke),
signed API requests and redirects, AccessToken building and
serializing, Response parsing (JSON, form, XML), errors, and the time
to require the gem. Every request goes to Faraday's test adapter, so
nothing waits on a socket.

The rest comes from the kettle-jem template, with workflows.duel set
in .structuredmerge/kettle-jem.yml: the Perf workflow, .duel.rb,
gemfiles/modular/duel.gemfile (loaded by the root Gemfile and a new
perf appraisal), the rake duel task, the RuboCop include for .duel.rb,
the README badge, and a CHANGELOG entry. Each file matches what
kettle-jem generates, so a template run keeps all of it, and nothing
needs a freeze block or a per-file override.

Duel requires Ruby 3.4 or later, so older Rubies install neither duel
nor stackprof, and rake duel only warns there.
Update the name in the license, the README, the gemspec's authors and
the generated docs.
Copilot AI lite review requested due to automatic review settings October 27, 2026 13:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Two unresolved moderate issues affect template safety and whether relevant performance changes trigger comparisons.

1 open issue
2 resolved since last review

Review effort: Lite

@sferik

sferik commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

@pboling I took you up on templating this branch myself. I ran kettle-jem install with kettle-jem from structuredmerge-ruby main (7.1.41, so not released yet) and workflows.duel: true. Then I rebuilt the PR on main with only the Duel parts of what it generated:

  • the Perf workflow, .duel.rb and duel.gemfile
  • the perf appraisal and the eval_gemfile line in the root Gemfile
  • the rake duel task and the RuboCop include
  • the badge and the CHANGELOG entry

All the overrides from the first version are gone: the freeze blocks, the per-file override in kettle-jem.yml and rakelib/duel.rake. A second template pass on top of the branch leaves all of it alone. Gemfile.lock only adds released gems: duel 0.1.1, fiddle and stackprof.

I left the template's other updates for your re-template of main. They include the nomono floors, K_SOUP_COV_MIN_STRICT, the sponsor and compatibility text in the README, and the coverage.yml change that skips the sticky comment for fork PRs. That last one is why "Code Coverage" fails here, since this PR comes from my fork. After you re-template main, a rebase should only conflict in Gemfile.lock and the CHANGELOG, and I'm happy to do it.

One thing I noticed: the template run also removed the "Direct sibling dependencies" block from the root Gemfile, which wires anonymous_loader, auth-sanitizer, snaky_hash and version_gem through RUBY_OAUTH_DEV.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants