Skip to content

Refactor Slowlog import stages and bound statistics memory - #63

Merged
somethingwithproof merged 2 commits into
fix/review-hardeningfrom
refactor/import-pipeline
Oct 24, 2026
Merged

somethingwithproof merged 2 commits into
fix/review-hardeningfrom
refactor/import-pipeline

Conversation

@somethingwithproof

@somethingwithproof somethingwithproof commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Imports now report failures to their callers and retain source files when ingestion or post-processing fails. The follow-up separates the streaming parser, batch writer, SQL table extractor, page queries and statistics, while keeping the public helper functions available.

  • Check import database reads and writes, return nonzero CLI failures, and delete input only after successful inline processing.
  • Reset optional parser fields between entries and bound detail batches by row count and serialized size.
  • Pass chart and detail filters explicitly to query helpers.
  • Share a statistics sample budget across groups and release each scope after writing it. Counts, totals, minima and maxima remain exact; percentile estimates are labeled in the chart view.
  • Run the existing integration workflow for stacked PRs and include every extracted helper in the existing coverage check.

Stacked on #60; this PR targets fix/review-hardening so the refactor can be reviewed separately.

Validation: 534 tests pass, including the real database and HTTP suite; CI changed-line coverage is 100% (792/792 lines). Real MariaDB regressions verify retained input and nonzero file-import, post-processing and reprocessing failures. PHP syntax, manifest, translation-template and workflow checks pass. CI passes on PHP 8.2, 8.3 and 8.4.

Five-run isolated PHP benchmarks show statistics peak allocation falling from 30 MiB to 14 MiB, with median time falling from 0.498 s to 0.459 s. Ingestion takes about 9% less time and table extraction is essentially unchanged. Database adapters are stubbed; these are not whole-application latency measurements. Reproduction commands and raw results are in tests/benchmarks/.

Comment thread includes/statistics.php Dismissed
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
@sonarqubecloud

sonarqubecloud Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.8% Duplication on New Code

See analysis details on SonarQube Cloud

@somethingwithproof
somethingwithproof merged commit 9b4b03f into fix/review-hardening Oct 24, 2026
5 checks passed
@somethingwithproof
somethingwithproof deleted the refactor/import-pipeline branch October 24, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants