Skip to content

fix: validate solver inputs and benchmark results - #398

Merged
bardhh merged 3 commits into
mainfrom
fix/solver-benchmark-validation
Sep 20, 2026
Merged

bardhh merged 3 commits into
mainfrom
fix/solver-benchmark-validation

Conversation

@bardhh

@bardhh bardhh commented Sep 20, 2026

Copy link
Copy Markdown
Owner

QP calls with incomplete constraint pairs could silently solve a different problem, and the fast solver could report success for an infinite unconstrained solution. Validate dimensions and constraint pairs across the adapters, including during JIT tracing, and reject non-finite success while preserving existing backend failure codes and error messages.

Benchmark reporting now rejects missing required metrics, unknown comparison/objective metrics, empty experiments, and mismatched batch result counts. Explicitly unavailable measurements remain null in artifacts and N/A in the sweep display instead of becoming zero violations. Sweep objectives and safety decisions require measured, finite values.

Remove obsolete tutorial migration tests and commented solver scaffolding, add active equality/infeasibility coverage, and extend Ruff checks to scripts and optimization/benchmark tests in both CI and pre-commit. Document the benchmark result contract.

Validation:

  • Core suite on Python 3.12: 868 passed, 8 skipped; additional status-preservation regressions passed separately.
  • MuJoCo suite on Python 3.10: 206 passed, 3 optional oracle tests skipped, 28 high-memory G1 tests excluded.
  • Examples and slow tests: 67 passed.
  • Targeted solver/benchmark tests passed on Python 3.10 and 3.12.
  • Ruff, Black, isort, mypy, pre-commit configuration validation, and diff checks passed.

Compatibility: incomplete benchmark records must now declare required keys. Use None for unavailable measurements; consumers must handle null summaries. Malformed QP inputs now fail explicitly instead of dropping constraints.

@bardhh
bardhh merged commit 83e9d65 into main Sep 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant