Skip to content

feat: Parallelizing test generation phase - #86

Merged
araujof merged 6 commits into
mainfrom
hl/generation_speed
Sep 30, 2026
Merged

araujof merged 6 commits into
mainfrom
hl/generation_speed

Conversation

@dhl123

@dhl123 dhl123 commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Summary

smith --flag test_generation runs four LLM stages — decompose → grey_condition → variable_extraction → case_generation. With BATCH_PROCESSING=true, each split its input into batches and sent one gateway request per batch in a serial loop, waiting for each response before issuing the next.

Now they are parallelized exactly as translation does. This applies the pattern established in extract_tool_args.py to the generation side.

  • Output order is preserved.
  • GENERATION_CONCURRENCY=1 (or -1) reproduces the old serial path exactly — no pool is created
  • Concurrency above the batch count is harmless — max_workers is a ceiling, and threads are created lazily

Closes: #84

Changes

File Change
src/smith/test_generation/concurrency.py New — resolve_generation_concurrency() + shared run_batches()
src/smith/test_generation/decompose.py Serial loop → run_batches; intra-batch pairing moved inside the worker
src/smith/test_generation/grey_condition.py Serial loop → run_batches
src/smith/test_generation/variable_extraction.py Serial loop → run_batches
src/smith/test_generation/case_generation.py Serial loop → run_batches
src/smith/cli.py Resolve GENERATION_CONCURRENCY once, thread it through generate_test
tests/integration/fakes.py FakeOpenAI thread safety + content-keyed by_prompt mode
tests/integration/test_generation_unit.py +9 tests
.env_template GENERATION_CONCURRENCY=4
docs/content/docs/configuration.md New row; fixed the TRANSLATION_CONCURRENCY default inconsistency
CLAUDE.md Configuration-model prose
CHANGELOG.md modified ### Added entry

Checks

  • make ci passes (lint, Rego lint, license headers, build smoke)
  • make test passes (policy scorecard — needed if policy behavior changed)
  • CHANGELOG.md updated under ## [Unreleased] (if user-facing)
  • PR title uses a conventional prefix (feat:, fix:, test:, docs:, chore:, or refactor:)
  • Commits are signed off for the DCO (git commit -s)

Notes (optional)

Design sketch, screenshots, or extra context.

Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123
dhl123 marked this pull request as ready for review September 29, 2026 15:15
@dhl123
dhl123 requested a review from araujof as a code owner September 29, 2026 15:15
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@araujof

araujof commented Sep 30, 2026

Copy link
Copy Markdown
Member

@dhl123 can you please merge main and resolve conflicts?

Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>

@araujof araujof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@araujof
araujof merged commit 8797aa4 into main Sep 30, 2026
9 checks passed
@araujof
araujof deleted the hl/generation_speed branch September 30, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Parallelizing test generation phase

2 participants