skills/ce-retune/references/noise-floor.md describes p^N as an exact test after estimating the baseline from A/A and archive runs. It also permits another streak after changing a failed candidate. The user guide repeats the one-armed-test claim.
For a fixed known p, independent runs and one pre-specified attempt, the calculation is valid. It is not automatically a significance level for a selected winner after repeated attempts, and an independently estimated baseline still has uncertainty.
A numerical counterexample (not measured CE performance): with p=0.58 and N=8, one planned all-success attempt has probability 0.0128063. Across 20 independent attempts under that same null, the probability of at least one success is 1-(1-p^N)^20 = 0.227235. Changing the candidate does not erase earlier selection opportunities.
Keep the operational streak bar, A/A measurement, fixed-build confirmation and stop-on-failure behavior. Qualify the statistical interpretation, retain all attempts, and distinguish candidate-selection evidence from a final check whose results were not used to select the candidate. When the assumptions or overall error control are unsupported, report the observed confirmation without an exact-significance claim.
Scope: the streak guidance and its user-facing explanation only; no new statistical engine, dependencies, fixed sample minimum, or changes to ce-optimize.
skills/ce-retune/references/noise-floor.mddescribesp^Nas an exact test after estimating the baseline from A/A and archive runs. It also permits another streak after changing a failed candidate. The user guide repeats the one-armed-test claim.For a fixed known
p, independent runs and one pre-specified attempt, the calculation is valid. It is not automatically a significance level for a selected winner after repeated attempts, and an independently estimated baseline still has uncertainty.A numerical counterexample (not measured CE performance): with
p=0.58andN=8, one planned all-success attempt has probability0.0128063. Across 20 independent attempts under that same null, the probability of at least one success is1-(1-p^N)^20 = 0.227235. Changing the candidate does not erase earlier selection opportunities.Keep the operational streak bar, A/A measurement, fixed-build confirmation and stop-on-failure behavior. Qualify the statistical interpretation, retain all attempts, and distinguish candidate-selection evidence from a final check whose results were not used to select the candidate. When the assumptions or overall error control are unsupported, report the observed confirmation without an exact-significance claim.
Scope: the streak guidance and its user-facing explanation only; no new statistical engine, dependencies, fixed sample minimum, or changes to
ce-optimize.