Skip to content

feat(bench): google benchmark against the gpt solver - #2

Merged
codethinki merged 3 commits into
masterfrom
feat/google-benchmark
Oct 6, 2026
Merged

codethinki merged 3 commits into
masterfrom
feat/google-benchmark

Conversation

@codethinki

@codethinki codethinki commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Replaces the hand written timings with a Google Benchmark target, lbs_solver_bench.

puzzles

  • assets/nyt_puzzles.txt: the 558 nyt puzzles from 2025-03-25 to 2026-10-05, one <date> <letters> per line
  • taken from the daily archive in cbrady2718/letterBoxedSolver (nyt_data.json), sides only
  • split into solvable and unsolvable with our word list (556 / 2)
  • lbs and gpt have to agree on solvability for every puzzle, otherwise the benchmark exits with 1

benchmarks

  • every puzzle is its own benchmark, so warmup, iteration count and DoNotOptimize are handled by Google Benchmark
  • lbs/load, lbs/split: the word list once
  • lbs/filter_and_hash, lbs/solve, lbs/total: per puzzle, word list already in memory (the #embed case)
  • lbs/total_with_load, gpt/total_with_load: per puzzle including reading the word list
  • a custom reporter prints min, p50, p90, p99, max and mean per benchmark, --benchmark_out has the per puzzle data
  • --benchmark_filter=2026- etc. selects puzzles by date
  • default min time is 0.01s per puzzle, the full run takes about 50s

other

  • gpt_runner.hpp moved into bench and reads the file with std::ifstream instead of a windows file mapping
  • benchmark>=1.9.5, the 1.9.0 port at our baseline passes -Werror=old-style-cast to cmake which cmake 4 rejects
  • runner and solver.real_world no longer time anything, real_world checks that a solution is found

results (solvable puzzles, p50 / p99 in us, Core Ultra 9 285H)

gcc 16 clang
lbs/split 1127 473
lbs/total 290 / 504 298 / 664
lbs/total_with_load 1357 / 1642 822 / 1174
gpt/total_with_load 962 / 1110 983 / 1458

@codethinki
codethinki merged commit 4fbf1ad into master Oct 6, 2026
6 checks passed
@codethinki
codethinki deleted the feat/google-benchmark branch October 6, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant