Skip to content

test: add compatibility discovery harness and revise SemVer proposal - #398

Open
Pino de Candia (pinodeca) wants to merge 3 commits into
mainfrom
pinodeca-semantic-versioning-guarantees
Open

Pino de Candia (pinodeca) wants to merge 3 commits into
mainfrom
pinodeca-semantic-versioning-guarantees

Conversation

@pinodeca

@pinodeca Pino de Candia (pinodeca) commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Extend this PR from a SemVer proposal into a compatibility-discovery foundation: reproducible released-binary B1/B2 tests, initial measured findings, and a proposal that keeps future guarantees separate from current evidence.

  • Add an isolated PostgreSQL runner that builds v0.2.2, v0.2.5, v0.2.7 and the 0.2.9 candidate with recorded source/package hashes, lockfiles, toolchain information and diagnostic evidence.
  • Run a single staged chain, alternating binary replacement on the previous catalog (B1) and ALTER EXTENSION UPDATE to the matching catalog (B2).
  • Start one finite SQL operation and one independently timer-paced root loop at each of seven states: fourteen instances total. Check progress, retained results, side effects, engine/df status and monitoring APIs across subsequent states.
  • Keep the original automatic A/B1/B2 suite intact. Add automatic harness unit tests; make the full released-binary chain a default-off manual CI option with evidence uploads.
  • Update docs/upgrade-testing.md with invocation, interpretation, evidence and coverage limits. Retain the historical audit and future versioning/guarded-upgrade proposals in docs/semver-compatibility-plan.md, but defer adoption of a 1.0 release or guaranteed replay baseline.

Tested Chain

State Binary Extension catalog Scenario
0 0.2.2 0.2.2 Baseline
1 0.2.5 0.2.2 B1
2 0.2.5 0.2.5 B2
3 0.2.7 0.2.5 B1
4 0.2.7 0.2.7 B2
5 0.2.9 candidate 0.2.7 B1
6 0.2.9 candidate 0.2.9 B2

Loop iterations advance independently of these states. Existing failed instances remain inspectable and are not counted as passing later continuity checks.

Initial Findings

Measured locally on Linux/PostgreSQL 17.10 with debug pg17-only replay artifacts and candidate runtime source at e175a2a9f6b72904ea6b04acce1c968496063775 (this PR's subsequent changes do not alter runtime source):

  • All seven finite instances completed with the expected result and one insertion; retained results remained readable at later states.
  • The loop created on 0.2.2 failed at the 0.2.5 binary replacement with an update-node-status recorded-input schedule mismatch. Provider status was failed while df.status() still reported running.
  • The six loops created at later states progressed through all applicable subsequent observations.
  • All fourteen instances remained inspectable through the checked monitoring APIs.

Strict mode returns exit 1 for the documented break. The explicit known-break option accepts only that fixture/transition/error combination; unrelated failures, diagnostic errors and incomplete coverage still fail. A green exception-enabled run is not a claim that every workflow survived.

Scope And Migration Impact

No runtime behavior, dependency versions, extension DDL, package version or public SQL API changes. No upgrade script or runtime schema detection is needed for this implementation.

These two graph shapes do not establish blanket feature compatibility, direct skipped-version upgrades, a v0.2.8 replay baseline, or provider-version migration compatibility. The selected binaries all use duroxide-pg 0.1.34. Source-inferred historical risks remain distinct from measured results. Runtime compatibility repairs, broader fixtures, sanitized downstream pipeline samples, checkpoint reuse and mandatory SemVer release gates are follow-up work.

Validation

  • cargo build --features pg17
  • cargo clippy --features pg17
  • cargo fmt -p pg_durable -- --check
  • ./scripts/test-unit.sh: 391 passed, 16 ignored
  • ./scripts/test-e2e-local.sh: 65 passed, 0 failed across all required modes
  • Original ./scripts/test-upgrade.sh --pg-version 17: all 142 A/B1/B2 checks passed
  • Replay harness unit tests: 20 passed, including timeout reporting, diagnostics/cleanup and invalid shell arguments
  • New shell entry point: all seven replay states completed with the sole documented known break
  • Strict-mode negative control: expected exit 1, all evidence retained, no unexpected failures
  • Shell syntax, YAML parsing/manual matrix routing, local documentation links and staged whitespace checks

PostgreSQL validation used private installations/clusters rather than the normal developer database. The original suite's fresh-cluster superuser setting was enabled only in its private test configuration. Remote CI results are separate from these local results.

Manual Replay Run

./scripts/test-upgrade.sh --pg-version 17 \
  --replay-chain "$PWD/target/replay-evidence" --allow-known-replay-breaks

Omit the allowance for strict mode. In CI, enable the replay_chain workflow-dispatch input and select PG17 or 17, 18; PG18-only with replay requested is rejected.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pinodeca
Pino de Candia (pinodeca) force-pushed the pinodeca-semantic-versioning-guarantees branch from 6dfafaf to e175a2a Compare September 23, 2026 20:05
@pinodeca
Pino de Candia (pinodeca) marked this pull request as ready for review September 23, 2026 22:32
@pinodeca Pino de Candia (pinodeca) changed the title docs: propose semantic versioning guarantees and compatibility plan test: add compatibility discovery harness and revise SemVer proposal Sep 24, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant