Repository navigation
[python] Generated output differs between identical runs without --embed-auditing-info #412
Description
Summary
Running the same nnvg --target-language=py command twice produces different files, even without --embed-auditing-info. Generated output is supposed to be identical when that flag isn't given (#324 reversed the flag so that reproducible output is the default). C output is identical between runs; Python output is not.
Reproduce
On main at ed853f9, in ghcr.io/opencyphal/toxic:tx26.4.1:
PRDT=submodules/public_regulated_data_types/uavcan
nnvg $PRDT --target-language=py --outdir a
nnvg $PRDT --target-language=py --outdir b
diff -rq a b | wc -l # 244 of 245 files differ (only nunavut_support.py matches)The same two runs with --target-language=c produce 0 differing files out of 176. C++ was not checked.
Causes
There are three independent causes. All three must be fixed for the output to be reproducible.
- The namespace file's timestamp ignores
--embed-auditing-info(affects all 69__init__.py).src/nunavut/lang/py/templates/Namespace.j2:8writes# Generated at: {{ now_utc }} UTCunconditionally. Python's type modules (py/templates/base.j2) and the C and C++base.j2templates write it only inside{%- if nunavut.embed_auditing_info %}. - The embedded model blob records the time (affects all 175 type modules).
filter_pickle(src/nunavut/lang/py/__init__.py:355) callsgzip.compress(pickle.dumps(x, protocol=4)). Without anmtimeargument, gzip writes the current time into its header, so each module's_MODEL_constant changes on every run. Decoding the blobs from both runs gives identical pickled payloads. Only the 4-byte gzipMTIMEheader field differs (for example1792505892against1792505915). Nothing else in the type modules differs. - Namespace file import order is nondeterministic (affects 30 of 68
uavcan__init__.py). Ignoring the timestamp line, 30 namespace files still differ, always in the order of theirfrom … import …lines and never in which types they list.Namespace.get_nested_types()(src/nunavut/_namespace.py:904) returns types in insertion order, and insertion order follows the order the DSDL files were read (set iteration plus parallel parse-job completion).
Expected
Without --embed-auditing-info, running the same command twice produces byte-for-byte identical Python output, as it does for C. With the flag, volatile metadata such as the timestamp may appear, as it does in the other languages.
Possible fixes
- Wrap the
Generated atline inpy/templates/Namespace.j2in{%- if nunavut.embed_auditing_info %}, matchingpy/templates/base.j2. - Pass
mtime=0togzip.compressinfilter_pickle. Consumers (_restore_constant_) only callgzip.decompress, which ignores the field. - Emit namespace members in a stable order, e.g. sorted by full name and version, in
get_nested_types()or in the template.
A regression test could generate the same input twice into two directories, without --embed-auditing-info, and assert that the trees are identical, for every target language.
Found while working on #403. It doesn't block #403, which needs semantic idempotence, not byte-for-byte reproducibility.
Activity