Skip to content

perf(runs): leave an unchanged imported job as it is on each workspace open - #264

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:perf/import-unchanged-source
Nov 20, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:perf/import-unchanged-source

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #259

What

Each open of a workspace converts its ATIF and OpenTelemetry roots again (shared._import_roots). Both importers rewrote every file of the converted job, so the job got new mtimes and workspace.load_job parsed it again, even when the source had not changed.

  • importers.write_if_changed writes a file only when its bytes differ from the file on disk, and says whether it wrote. The ATIF importer uses it for the job's result.json and config.json, each trial's result.json, and the trajectory.json copy (still byte for byte). The OpenTelemetry importer's _dump uses it for the same files and trace.json.
  • importers.mark_changed moves the job's result.json mtime when any trial was written. An ended job's cache is checked by the job's own records and folder, and a change inside one trial can leave those identical, so without this an edited source would keep its old parse.

Comparing the output bytes, rather than stamping the source, also covers a new importer version: any change in what it writes is written. A trial whose source is gone is removed as before.

Touches

backend (nullius/importers/__init__.py, nullius/importers/atif.py, nullius/importers/otel.py) · tests (tests/test_import_unchanged_source.py)

Verified

In a rootless Podman container (python:3.12):

  • python3 app/tests/parallel.py: 3106 tests in 158 modules, OK.
  • pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 555 tests in 56 modules, OK.
  • New app/tests/test_import_unchanged_source.py, one test per importer: convert, read, convert again over the unchanged source (no file in the job moves, and load_job returns the cached parse), then edit the source in a way that leaves the job's own records identical and convert again (files move, and the job is parsed again).
    • With the importers from main, both tests fail: config.json, result.json and each trial's files move, and the second read parses.
    • Without mark_changed, both fail on the edited source: the stale parse is returned.

Risk

  • Each conversion now reads each existing output file once to compare it. These are the files it wrote before, and the source is read whole already.
  • A file in imported/ edited by hand to the exact bytes the importer would write is left as it is. Any other edit is overwritten, as before.

…e open

Each open of a workspace converts its ATIF and OpenTelemetry roots again. The importers rewrote every file of the converted job, so new mtimes made workspace.load_job parse it again. They now write a file only when its bytes differ from what is on disk, so an unchanged source keeps the job's mtimes and its cached parse.

A changed source is still written and parsed again. An ended job's cache is checked by the job's own result.json, config.json and folder, which a change inside one trial can leave identical, so a written trial also moves the job's result.json mtime.
@harenome
harenome merged commit 872e60a into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the perf/import-unchanged-source branch November 20, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Imports: each open of a workspace rewrites its unchanged ATIF and OpenTelemetry jobs, so they are parsed again

1 participant