Skip to content

perf(runs): stop reading every trial of every job on each page - #215

Merged
harenome merged 8 commits into
ThinkFlowLab:mainfrom
harenome:perf/many-workspaces
Nov 19, 2026
Merged

harenome merged 8 commits into
ThinkFlowLab:mainfrom
harenome:perf/many-workspaces

Conversation

@harenome

@harenome harenome commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #214

What

Each change below makes one cause cheaper and keeps the numbers the same.

  1. An ended job is checked by its own records (workspace.load_job). A job's head is the schema, its folder's mtime, and its top-level result.json and config.json. A job that ended (finished_at set, state finished) with an unchanged head is read from memory or disk with no per-trial stats. Other jobs keep the full walk. The cache file now carries the head. An older cache learns it on its first rewrite, so no schema bump is needed. Refresh (clear_cache) stays the way to read a job again.
  2. A single job's pages read its own folder (shared._job_for_request). Trial, trace, live, judge, judge tasks, reconcile and the builder's ?like= find the one folder named for the job. Any doubt falls back to today's full load: ?workspace=, a miss, or several folders of that name. A run's own page keeps the full load, because its Delete check reads the whole workspace anyway.
  3. Each run record's jobs folder is resolved once per request. Workspaces resolves each workspace's roots once (_real_roots), through a per-request memo, shared.request_memo.
  4. Launches sizes an ended run only when its folder moves (execution.run_size). A run in progress is measured at each load. An ended run's size is cached under a lock, keyed on its state, its finish time, and the mtimes of its folder, run.json and jobs/. deletion() still measures afresh, so both Delete prompts show the current size.
  5. Run records are indexed by job once per request (execution.owners()). One dict maps each job to the newest record that names it, with the first one listed winning a tie, as max(key=started) does. for_job and ended_for take it. Launches used a different tie-break before, so on equal start times it could name a different run than the Runs page. Now both agree.
  6. Outcome counts are read once per row. runs/index.html, runs/job.html, compare/compare.html and runs/feasibility.html read counts once into a variable. Job.counts stays a property: trials are added after a job is made, so a cached count could go stale.
  7. A job that a run is still writing is checked trial by trial. A host run's folder comes back by rsync. A pass can copy a trial's result.json while it is being written, in the same pass that brings the job's final result.json. The final pass then fixes that trial file only, and the head does not move. So load_jobs takes the owning run's state: none while the run is alive, which turns the shortcut off, and its state and finish time once it ends, which joins the head. The job gets one full check after the last copy. The final copy runs before execution.py records the run's end. A part's own end is recorded before that copy, for example under Stop, so it counts only once the whole run has ended (execution.run_ends). The Delete prompt reads jobs with the same states. A job opened from a folder has no record and takes the shortcut as before. workspace.py still imports nothing from execution.

Not done: Workspaces still counts each workspace's launches from its parsed jobs. Exact counts need, for every view, its id, merges, state, launch, dataset, start and check role, which is the parse cache again. After the commits above, the page takes 0.7 s.

Touches

backend (nullius/workspace.py, nullius/execution.py) · routes (web/shared.py, web/runs.py, web/traces.py, web/judge.py, web/judge_tasks.py, web/reconcile.py, web/builder.py) · templates (runs/index.html, runs/job.html, runs/feasibility.html, compare/compare.html) · tests

Verified

A synthetic profile in a rootless Podman container (python:3.12): 100 workspaces, 2000 Harbor jobs of 50 trials, 200 run records, 480 jobs in the open workspace. Each time below is the median of requests 2 to 5 over two fresh processes, in seconds. Runs vary by about 30%.

Page main this branch
Workspaces 4.77 0.69
Launches 2.15 0.50
Runs 1.44 0.46
A run 0.60 0.07
A trial 0.59 0.04
Compare, 2 runs 0.59 0.07
A benchmark 0.59 0.07
Run builder 0.68 0.19
Settings 0.04 0.04
  • Same output: /runs, /runs?view=all, /workspaces, /launches, a run page and a trial page were saved on main and on the branch, on the same data at two sizes (500 and 2000 jobs). After normalising the CSRF tokens, the interface digest and the workspace's "opened" time, they are identical. No count and no size differs.
  • python3 app/tests/parallel.py: 2325 tests in 130 modules, OK.
  • pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 644 tests in 54 modules, OK.
  • New tests:
    • test_workspace.py: an ended job is not stated trial by trial, from memory or disk; a resume and a new trial folder read the job again; a running job updates each trial; an old cache learns its head once; the rsync race reads the corrected trial once the run ends, and the test fails without the guard; a folder-opened job takes the shortcut.
    • test_execution_stop.py: a part stopped as it ends gives no end until the run has ended, and the test fails when a part's end counts alone.
    • test_job_lookup.py: one folder, ?workspace=, two folders of one name, and a miss.
    • test_run_delete.py: an ended run's size is cached and read again when its folder moves; a Delete prompt measures afresh; a running run is measured at each load.
    • An index test with five records, overlapping jobs and a tie, against the old expression.

Not checked: a real host run that hits the race, and the docker test tier.

Risk

  • The shortcut trusts a finished job's head. A hand edit of one finished trial's file in a folder Nullius did not launch shows only after Refresh. That is the documented limit, and the one test that edited a trial in place now edits the shard's own result.json.
  • The first start on this branch rewrites each cache once to add its head. On the profile above, the first Workspaces load took 23 to 34 s in that process, then 0.7 s.
  • Launches can show an ended run's old size if a file changes deep in its folder and none of the cached mtimes move, for example a trace view saved later. Delete prompts are not affected. The code marks this with a shortcut: comment.
  • To undo, revert the commits. The cache's added head field is ignored by main.

A job whose result records a finish with nothing running or pending is
read from the cache while its result.json, config.json and folder mtime
are unchanged, without stating every trial's files. A resume rewrites
result.json and a trial folder added or removed moves the folder, so both
are still parsed again; a running job keeps the full signature. Refresh
reads every job again.
The trial, judge, reconcile, live-board and builder-like lookups found
one job among every job of the workspace. When the open workspace has
exactly one folder of that job's name, it alone is read; another
workspace asked for, no such folder or several read the whole workspace
as before. The job page keeps the whole walk: its Delete groups every
launch of the workspace anyway.
The Workspaces page matched every run record against every workspace,
resolving the record's folder and each root again for each pair. Each
workspace's roots are now resolved once, and each record's folder once
per request.
The Launches page walked every run folder on each load. A run still
alive is walked at each ask; an ended one again only when its state, its
record, its folder or its jobs folder moved. Delete's question walks the
folder afresh.
The Runs page asked for_job of every job and launch, each a scan of every
run record. execution.owners indexes them once, with for_job's tie-break:
the newest start, the first listed of a tie. The Runs page reads it once
per request, and the Launches page's own index is the same one, so the two
cannot name different runs for a job.
Job.counts walks every trial on each read, and the Runs list read it up
to a dozen times per row. The templates that read a job's counts, or a
feasibility report's, more than once now read them once into oc. The
counts stay a property: a job's trials can change after it is made.
A host run's copy can bring a job's final result.json while a trial's
file is still mid-write; the next copy corrects the trial alone, which
the head of an ended job does not see. While a run of this application
writes a job, the job is checked trial by trial, and its head carries
the run's end, so the last copy is checked once more. The web layer
passes each run's job_state; a job no run writes is unchanged.
A part's end is written down before the run's last copy from its host,
for example under Stop. Counted alone, it let a page cache a job whose
trial that copy then corrected. The Delete prompt reads jobs with the
same run states.
@harenome
harenome force-pushed the perf/many-workspaces branch from e35e79d to 9de40ff Compare November 19, 2026 09:56
@harenome
harenome merged commit f6ef315 into ThinkFlowLab:main Nov 19, 2026
2 checks passed
@harenome
harenome deleted the perf/many-workspaces branch November 19, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pages get slow with many workspaces, runs and trials

1 participant