Skip to content

Python: merge namespace files instead of replacing them (#403) - #415

Merged
thirtytwobits merged 3 commits into
mainfrom
issue403-append-only-namespaces
Oct 23, 2026
Merged

thirtytwobits merged 3 commits into
mainfrom
issue403-append-only-namespaces

Conversation

@thirtytwobits

@thirtytwobits thirtytwobits commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Fixes #403. Supersedes #410, which was abandoned.

Generating Python into an output directory that already held generated code replaced each package's __init__.py with only the current run's types. Types written by earlier runs could then no longer be imported from their package.

The change

Namespace files are now append-only for languages that set the new merge_namespace_files property. Only Python sets it.

  • Each namespace file ends with a manifest comment. It records the file's namespace, its types, and a digest of how they were named.
  • A later run adds its types to those recorded and recomputes the newest-minor-version aliases. Generation never deletes a file.
  • The manifest is read without regard to whitespace, line breaks or comment characters, so formatters don't affect it.

Before writing anything, generation refuses in four cases (NamespaceConflictKind), and writes nothing:

Kind When --force
NO_MANIFEST An existing namespace file has no valid manifest Writes only this run's types
NAMING_CHANGED This run would name the recorded types differently Writes only this run's types
CASE_COLLISION Two types' files would differ only in letter case Can't be forced
NAMESPACE_COLLISION Two namespaces would share one namespace file (case, or stropping such as import/import_), within a run or across runs Can't be forced

nnvg lists every refused file and exits 1. --dry-run, --list-outputs and --list-inputs report the same refusals. The API raises NamespaceConflictError.

The guarantees are deliberately weak:

  • Output is defined only when Nunavut alone modified it.
  • clean → a → b → c within one build is the supported workflow.
  • Generating across builds without a clean is discouraged.

Upgrade impact (release note)

  • The first run refuses once. The first run into an output directory written by an older Nunavut refuses with NO_MANIFEST. Clean the directory, or pass --force once.
  • Custom Python namespace templates must write the manifest with the namespace_manifest_lines filter (see the template guide). Otherwise every later run into the same directory refuses.
  • Imports in Python namespace files are now sorted (review finding R13). This also helps reproducibility.
  • API: generate_all, generate_all_for_language, generate_all_from_namespace and generate_all_from_namespace_with_generators gain a trailing force=False parameter. Existing positional calls are unchanged.
  • Same-run namespace collisions are now refused. They used to silently drop types on main.

Gated process artifacts (all approved on #403)

  • Problem statement
  • Design: includes the input-enumeration and failure-mode tables.
  • Test plan
  • Amendments:
    • 1: a stropping change relocates a namespace.
    • 2: list-only runs, --no-overwrite, manifest placement.
    • 3: namespace identity, NAMESPACE_COLLISION.

The design and test plan comments are updated in place for each amendment. G3 and V1 were shown failing on main before the fix.

Self-review and independent reviews

  • Self-review: the input-enumeration and failure-mode tables were walked against the code, and the diff was checked against the design.
  • Independent review 1 (fresh agent: issue, design and diff only) found 14 items:
# Root cause Class Change
R1, R2 Existing run modes (--list-*, --no-overwrite) weren't considered together with the new refusal Each input combined with each refusal condition Amendment 2; tests G38, G39
R4 Namespace-level case collisions weren't in the design Namespace identity (reopened by review 2) Amendment 2, replaced by Amendment 3
R3 A crash after the manifest was written could leave a complete manifest on a truncated file Write ordering Manifest written last; test G42
R5–R10 Test gaps: shared consistency check, dry-run vs real run, split manifest lines, recorded-only keyword and service members, positional calls, rerun hygiene Tests not derived from every row Tests added or extended
R11, R14 Docs: example messages, links, heading level Docs Docs updated
R12 The verification noxfile reused its output directory Generating into a non-clean directory The noxfile generates into a clean directory
R13 Imports are now sorted Behavior change Noted above
  • Independent review 2 (fresh agent) found N1–N7:

    • N1 and N3 share a root cause with R4. Namespace identity was never recorded: the R4 fix compared the namespaces of members, not the namespace itself. That missed namespaces with no types of their own, and it broke idempotence for mg.import/mg.import_.
      • Class: two DSDL namespaces whose namespace files are the same file. The instances are case-only names and stropping pairs, within a run and across runs.
      • Fix: Amendment 3. Manifests record their namespace, and NAMESPACE_COLLISION can't be forced. Tests U8, G45–G47 and A6.
    • N2: inaccurate messages; each conflict kind now has its own message and advice.
    • N4–N7:
      • N4: the consistency check now runs after every Python generation, with named exceptions.
      • N5: --list-inputs is tested too.
      • N6: the slow-marker lookup uses get_closest_marker.
      • N7: G11's cleanup works on Windows.
  • Independent review 3 (fresh agent) found no code defects, and four low findings, all fixed:

    • stale docstrings;
    • test G48 (a stropping collision across runs);
    • the test plan's list of consistency-check exceptions;
    • a docs note on stale namespace collisions.
  • CI and SonarCloud on this PR (fixed in 8a78313):

    • The main suite failed on Python 3.14+ in CI (156 tests, FileNotFoundError connecting to the forkserver).
      • Root cause: the noxfile's --basetemp was nox's tmp directory, which nox also exports as TMPDIR. pytest empties a given basetemp the first time a test uses tmp_path. On 3.14+, the forkserver (now Linux's default start method) keeps its socket in TMPDIR, so that socket was deleted mid-run.
      • Why it showed up only now: no test on main uses tmp_path.
      • Why local runs missed it: the local main-suite runs were on 3.11 only, at a path too long for a Unix socket, so CPython fell back to /tmp.
      • Class: a pytest session whose --basetemp is TMPDIR.
      • Instances: the test, doctest and rstdoctest sessions. All three now use a pytest subdirectory. verification/python passes no --basetemp, so it isn't affected.
      • Verified: the CI failure was reproduced in CI's container with the old noxfile (78 failed, 215 passed). With the fix, the main suite passes on 3.14 and 3.15.
    • SonarCloud S6546 (one Union hint) and S9073 (three composite assertions): every instance in the diff is fixed, and the gate shows 0 new issues.
    • Codex found no issues.
  • Maintainer review: the standalone docs page was removed (b3040a4) because it made a minor consideration a top-level concept. The template guide keeps a short section on writing a namespace template that merges, and --force's help describes the refusals.

  • Process fixes, which belong to the maintainer's process draft (not part of this PR):

    • enumerate each input together with each refusal condition;
    • write tests that don't depend on their environment;
    • write a finding's class and every instance down before changing code;
    • carry every amendment through every artifact;
    • run the main suite on the newest supported Python as well as the oldest, from a short native path. The test plan's suite list is updated to match.
  • Defects caught locally, before review:

    • G33 assumed os.access reflects permission bits, which isn't true as root.
    • G40 broke on a macOS directory mounted into a Linux container.

    Both are now environment-independent.

Suites run

Run in ghcr.io/opencyphal/toxic:tx26.4.1 (linux/arm64) on the final code:

Suite Result
nox -s test-3.11 (with NUNAVUT_RUN_SLOW_TESTS=1) 294 passed, including slow G11 and all namespace-merge tests
doctest-3.11 / rstdoctest-3.11 379 / 7 passed
test-3.15, test-3.14 (checkout on the container's own filesystem, at 8a78313) 293 passed each (G11 is skipped; it runs in the main CI job)
doctest-3.15 / rstdoctest-3.15 379 / 7 passed
lint, docs passed
verification/python (nox) 731 passed on each of 3.11, 3.12, 3.13, 3.14 and 3.15, including V1 and V2; black passed

Not verified locally (CI covers these):

  • amd64 and native32;
  • the Windows and macOS compatibility jobs, the only case-insensitive native filesystems;
  • the C/C++ verification workflows (no C/C++ code changed);
  • the x86-only fuzz jobs.

🤖 Generated with Claude Code

Generating into an output directory that already holds generated Python
code replaced each package's __init__.py with only the current run's
types, so types written by earlier runs could no longer be imported
from their package.

Namespace files are now append-only for languages that set the new
merge_namespace_files property (Python only). Each namespace file ends
with a manifest comment recording its namespace, its types, and a digest
of how they were named. A later run adds its types to those recorded,
recomputes the newest-minor-version aliases, and never deletes a file.
The manifest is parsed without regard to whitespace or comment
characters, so formatters don't affect it.

Before writing anything, generation refuses if an existing namespace
file has no valid manifest, if this run would name the recorded types
differently, if two types' files would differ only in letter case, or
if two namespaces would share one namespace file. The new --force flag
(force=True in the API) writes the first two kinds with only this run's
types; the last two can't be forced. nnvg reports every refusal and
exits 1; the API raises NamespaceConflictError.

Imports in Python namespace files are now sorted. The verification
noxfile generates into a clean directory, and a new slow pytest marker
keeps the exhaustive sequence test (G11) to the main CI test job.

Docs: a new page on generating into an existing output directory, plus
notes in the CLI, language and template guides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thirtytwobits

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T20:42:37.073687Z fb3c730 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: fb3c7309d9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

compat-test-python3-ubuntu failed on Python 3.15: every
multiprocessing.Pool after the first test that used tmp_path failed to
connect to the forkserver. The noxfile passed --basetemp equal to the
nox tmp directory, which nox also exports as TMPDIR. pytest empties a
given basetemp when a test first uses tmp_path, and on Python 3.14+ the
forkserver (now Linux's default start method) keeps its socket in
TMPDIR, so the socket was deleted mid-session. No test on main used
tmp_path, so this first showed with the namespace-merge tests. Local
runs hid it: their TMPDIR path was too long for a Unix socket, so
CPython fell back to /tmp.

Class: a pytest session whose --basetemp is the process's TMPDIR.
Instances: the test, doctest and rstdoctest sessions in noxfile.py; all
three now use a pytest subdirectory. verification/python passes no
--basetemp and isn't affected.

Sonar: use `bytes | str` for the one Union hint this change adds
(S6546), and split the three composite assertions in the new tests
(S9073); greps of the diff found no other instances of either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The maintainer judged that a top-level page elevated a minor
consideration to a top-level concept. Remove docs/namespace_merging.rst,
its toctree entry, the pointers to it from the CLI and language guides,
and the URL in the --force help. The template guide keeps its short
section on what a namespace template must do to merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@thirtytwobits
thirtytwobits merged commit af8dcf3 into main Oct 23, 2026
65 checks passed
@thirtytwobits
thirtytwobits deleted the issue403-append-only-namespaces branch October 23, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[python] Generating a dependent namespace overwrites the dependency's namespace __init__.py

1 participant