Skip to content

fix(conformance): refuse a provisional tag that is not a §12 reading - #24

Merged
CameronBrooks11 merged 4 commits into
mainfrom
fix/runner-refuse-unknown-provisional-tag
Oct 21, 2026
Merged

CameronBrooks11 merged 4 commits into
mainfrom
fix/runner-refuse-unknown-provisional-tag

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

conformance/run_cases.py accepted any non-empty string as a case's provisional tag. A tag naming a reading the owner has since settled passed silently. With "provisional": "Ids are ASCII letters, digits and _ (§4.4)." added to grammar/ids-ascii-forms (a settled reading), the old runner gives 322 passed … rc=0. This runner gives 321 passed, 1 malformed … rc=2.

Change

  • Where the list is. The runner reads the provisional list from SPEC.md's ## <n>. Provisional… section (§12). The heading is found by its title, not its number. A SPEC.md with no such heading, or more than one, is no verdict (exit 2).
  • What counts as a reading. A reading is a non-empty - bullet after that heading and before the next bold paragraph (**Settled by the owner…) or heading. Each bullet is stripped, and its two-space continuation lines are folded in with single spaces.
  • When a case is malformed. Its tag is not a non-empty string, or is not one of those readings. Now that docs(spec): settle the last three provisional readings #22 has merged, the list is empty, so no tag is admitted.
  • Docs. conformance/cases/README.md now says the list must keep that shape: plain - bullets with two-space continuation lines. Other shapes (a bold lead-in, a ### subheading, * bullets) give an empty list, which fails closed. Lazily wrapped or nested bullets would be misread, which is why the README states the constraint.

Tests

New tests in tests/test_run_cases.py:

  • a tag naming a reading the real SPEC.md lists as settled;
  • tags that are settled, truncated, unfolded or a bare section number;
  • an empty list;
  • an empty - bullet;
  • a bullet that needs stripping;
  • the list ending at the next heading;
  • a renumbered section (another section is now ## 12.);
  • a doubled section;
  • a missing SPEC.md, or one with no Provisional section;
  • load_expect refusing a non-string or blank tag even where the readings would admit it.

Each test that guards a fix failed before that fix:

  • the first round failed 10 tests, the settled-reading one with assert 0 == 2;
  • the second round failed 10 tests, on the empty bullet, the renumbered and doubled sections, and the restored non-empty check.

The stripping and heading-stop tests exist to kill mutants that used to survive.

Mutants added to just mutants, all killed:

  • runner-provisional-tag-unchecked
  • runner-settled-list-read-as-provisional
  • runner-list-ends-only-at-bold
  • runner-provisional-bullet-not-folded
  • runner-provisional-continuation-dropped
  • runner-provisional-bullet-not-stripped
  • runner-empty-bullet-is-a-reading
  • runner-provisional-heading-by-number
  • runner-two-provisional-sections-allowed
  • runner-no-section-12-is-a-pass
  • runner-provisional-any-string (restored)

Gates (head of branch, after merging main)

just check          rc=0  All checks passed!
just test           rc=0  430 passed
just conform        rc=0  322 case(s): 322 passed, 0 failed, 0 malformed; 0 provisional, 0 of them failed
just conform-exact  rc=0  322 case(s): 322 passed, 0 failed, 0 malformed; 0 provisional, 0 of them failed
just mutants        rc=0  128 mutant(s), 2 equivalent, 0 problem(s)

The runner accepted any non-empty string as a case's provisional tag,
so a tag naming a reading the owner has since settled passed silently.
It now reads SPEC.md §12's list of provisional readings, the bullets
after the §12 heading and before the next bold paragraph, with their
line breaks folded to spaces, and refuses a case as malformed when its
tag is not one of them. A SPEC.md with no §12 is no verdict.

The membership check also rejects an empty or blank tag, so the
separate non-empty test and its mutant go; a tag must still be a
string. New mutants cover the check, the end of the list, the folding
of continuation lines and the missing-§12 exit.

Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
…nown-provisional-tag

Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The runner found SPEC.md's provisional list by the heading number
"## 12.", so renumbering the sections would point it at another
section and admit that section's bullets as readings. It now finds the
one "## <n>. Provisional" heading by its title, and a SPEC.md with no
such heading, or more than one, is no verdict.

An empty "- " bullet no longer counts as a reading, and the check that
a tag is a non-empty string is back, so an empty tag is refused
whatever SPEC.md holds.

New tests cover an empty bullet, a stripped bullet, the list ending at
the next heading, a renumbered section and a doubled section, and call
load_expect directly so the non-empty test checks what its name says.
Each has a mutant in the mutation gate.

Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The runner reads SPEC.md §12's provisional list only as plain "- "
bullets with two-space continuation lines, ending at the first bold
paragraph or heading. Say so beside the provisional tag, so an edit
to §12 keeps the list in a shape the runner reads.

Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
@CameronBrooks11
CameronBrooks11 merged commit 9b7c89b into main Oct 21, 2026
7 checks passed
@CameronBrooks11
CameronBrooks11 deleted the fix/runner-refuse-unknown-provisional-tag branch October 21, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant