[benbuschmann]

Eight fixes for 0.29.0. Two of them matter most. A Jobs API submit or withdraw at the wrong moment could leave a request counted forever, which quietly stopped all scheduling until a restart. And a start_cmd that failed at the end of an exclusive window left the service down with no explanation. The other six cut long waits under the transition lock, make wrong-shaped state files harmless, and tighten the gate.

Each fix is its own commit, and each comes with a test that failed on main before the change. The failing output below is trimmed.

A request counted forever after a Jobs API change

What went wrong. request_start counted the request, then looped over the job table and read the queue file once per job. A submit or withdraw during that loop raised dictionary changed size during iteration. The gate called request_start outside its try, so request_end never ran and inflight stayed at 1. From then on the service never looked idle: no parks, no gap jobs, and paused yield jobs stayed paused. snapshot, status, _track_progress, _track_speed and sample looped the same way and could fail one read or one tick.

Fix. Those loops now run over list(self.job.items()), status skips a job withdrawn since its snapshot, and the gate calls request_start inside the try whose finally runs request_end.

Test. tests/test_job_table_changes.py

before:
E       AssertionError: 1 != 0
E           RuntimeError: dictionary changed size during iteration
E           KeyError: 'late'
3 failed
after:
3 passed

A failed start_cmd at the end of an exclusive window

What went wrong. Only a successful start set starting_since. After a failure the service was neither stopped, starting nor dead, so requests got no 503 reason, no gap jobs were admitted, and the restart in _recover began only after a real request had failed its wake.

Fix. The failure branch sets dead_since. Requests get "failed to wake…; it is being restarted", and _recover, later in the same tick, runs stop_cmd and start_cmd once. Keeping stopped = True as well would not have worked, because _recover skips stopped services and _windows would re-run start_cmd every tick.

Test. tests/test_window_start_fails.py::test_failed_start_at_window_end_is_reported_and_retried (the start_cmd fails once, then works)

before:
E       AssertionError: 'service_restart' not found in ['service_stop', 'service_start']
after:
1 passed

Stop and withdraw waited up to 15 s holding the lock

What went wrong. stop_job sent SIGTERM, then polled for up to 15 s inside with self.lock, and remove_job held the lock around the whole stop. A request for a sleeping service, the tick, and other actions waited behind it.

Fix. stop_job takes the lock to send SIGTERM, waits without it, and takes it again for the SIGKILL, the exit check and the state change. remove_job calls stop_job first, outside the lock. The hold that stop_job writes to the queue file before it looks for processes keeps the scheduler from starting the job in between, so the 0.26.3 guarantee still holds: a job that starts at that moment is stopped, not lost.

Test. tests/test_stop_without_lock.py: the lock can be taken while a Stop waits on a job that ignores SIGTERM, a tick in between doesn't restart it, and it's still killed at 15 s. The same lock check runs for a withdraw.

before:
>       self.assertTrue(lock_free_while(r, t.is_alive))
E       AssertionError: False is not true      (both tests)
after:
2 passed

A revoked governor app was waited for up to timeout_s

What went wrong. GovernorJob.park waited up to timeout_s (120 s by default) for a revoked app to step aside. Parks started by the tick hold the transition lock for that whole wait.

Fix. The wait is now min(timeout_s, grace_s + 10). The launcher kills its command grace_s after a revoke, so an app still there 10 s later isn't stepping aside, and kill_stuck_jobs takes over as before. Apps that step aside within their grace period see no change. The grace_s row in docs/CONFIG.md says so. This covers governor jobs only. restart jobs on the tick path keep their full timeout_s, because a shorter cap could SIGKILL a job while it saves a checkpoint on SIGTERM.

Test. tests/test_governor.py::RevokeWaitTest::test_a_revoke_the_app_ignores_waits_its_grace_not_timeout_s (a granted app that never steps aside, with timeout_s = 30 and grace_s = 0.5). It runs without /proc, so it runs on macOS too.

before:
E       AssertionError: 30.057213249994675 not less than 11.5
after:
1 passed (10.6 s)

State files with the wrong shape

What went wrong. The readers handled missing files and invalid JSON, but not valid JSON of the wrong shape. A WINDOW file such as {"until": "06:00"}, or a list, made every tick fail, and every request that had to wake a sleeping service lost its connection with no answer. A wrong-shaped queue.json did the same. A wrong-shaped costs.json, gaps-<service>.json or submitted.json stopped the daemon from starting at all.

Fix. A WINDOW file that isn't a window counts as no window (never as an open-ended one), and the tick logs it once per version of the file as window_file_ignored. read_queue returns an empty queue unless it gets an object with a jobs list, and skips rows that aren't objects. CostBook and GapModel.load also catch AttributeError and TypeError. In Submissions.load, the per-row checks moved inside the per-row try.

Test. tests/test_bad_state_files.py

before:
E       AttributeError: 'list' object has no attribute 'get'
E       ValueError: could not convert string to float: '06:00'
E                   AttributeError: 'list' object has no attribute 'get'
3 failed
after:
3 passed

A job's own request could be parked under

What went wrong. After deciding to park, the tick re-checked only inflight. A job's own request (/_job/<name>/…) counts in job_inflight, so one that arrived between the snapshot and the park went ahead and the service was put to sleep under it.

Fix. The re-check is now if s["inflight"] or s["job_inflight"], as _switch already does.

Test. tests/test_job_request_park.py::test_a_job_request_after_the_snapshot_keeps_the_service_hot

before:
E       AssertionError: 'parked' != 'hot'
after:
1 passed

Queue file changes could overwrite each other

What went wrong. The page's actions, Stop, max_hours, a frozen job's step-aside and Jobs API submit and withdraw each read queue.json, changed it and wrote it back with no lock. Two at once lost one change. A lost Stop hold meant the job started again. They also shared one temp file, so one writer could fail with FileNotFoundError.

Fix. A separate queue_lock (an RLock) now covers each read-modify-write: the queue edits in ui.act, _set_hold, add_job and remove_job. Stop and max_hours now set their hold through _set_hold. The lock is only taken after Runner.lock, never before it, and page actions don't wait behind parks.

Test. tests/test_queue_writers.py::test_concurrent_queue_changes_are_all_kept: 200 page edits race 200 holds set by the runner. It failed on every run before the change.

before (3 runs):
E       AssertionError: Lists differ: [0] != []
E       AssertionError: Lists differ: [0, 1, 2, 3, ... 199] != []   (thread died: FileNotFoundError on queue.json.tmp)
E       AssertionError: Lists differ: [0] != []
after (5 runs):
1 passed  x5

The gate trusted Content-Length and had no timeout

What went wrong. A non-numeric Content-Length dropped the connection with no answer. -1 read until EOF and hung. There was no size limit, and a chunked body was read as the next request. Without a socket timeout, a silent client held a thread for good.

Fix. The gate answers 400 for a non-numeric length, 413 above 64 MiB and 411 for a body sent with Transfer-Encoding, and closes the connection in each case. These requests are never counted. Gate sockets get timeout = 60, like the UI's. The wait for the engine still uses request_timeout_s.

Test. tests/test_gate_request_body.py

before:
E       AssertionError: None != 400
E       AssertionError: None != 60
2 failed
after:
2 passed

Full suite

On macOS. The 17 skipped tests need Linux /proc, as on main.

python3 -m pytest -q                                  166 passed, 17 skipped
python3 -W ignore -m unittest discover -s tests       Ran 183 tests ... OK (skipped=17)

main had 152 passed and 17 skipped under pytest. This branch adds 14 tests. No live hardware tests were run for this PR.