Skip to content

Guard shell log lifecycle across processes - #140

Merged
luciusmagn merged 3 commits into
lambda-symbolics:masterfrom
htayj:fix/shell-log-cross-process-lifecycle
Oct 28, 2026
Merged

luciusmagn merged 3 commits into
lambda-symbolics:masterfrom
htayj:fix/shell-log-cross-process-lifecycle

Conversation

@htayj

@htayj htayj commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Fixes #139

Problem

A second process sharing the data directory can report a live shell log as interrupted and delete its output under retention pressure. An allocation abandoned before its manifest is written can also block later shell commands.

Fix

  • with-shell-log-lock adds a process-shared lock around allocation, capture recording and release, including the existing quota scan and reservation.
  • shell-log--active-p recognizes live owners in other processes through ownership leases. shell-log-allocate acquires the lease before publishing the manifest; shell-log--release-owner releases it after capture completion or artifact release.
  • shell-log--scan-manifest lets pruning reclaim empty abandoned allocations and leftover sexp-store manifest staging files from interrupted publication, recognized by shell-log--manifest-staging-p, while refusing unexpected contents. shell-log--capture-file-p reserves the ownership filename, and shell-log--trim-tombstones removes it when the tombstone expires.

Tradeoffs

  • Each active capture holds one extra file descriptor. This lets the kernel release ownership after a crash; avoiding it would require a different ownership protocol.
  • All processes sharing the directory must be upgraded for the protection to hold. Older processes cannot honor a locking protocol they do not implement.

Tests

  • test-shell-log-cross-process-lifecycle: independent worker processes read live output without interruption or pruning, reclaim storage after owner exit, remove expired ownership files and recover from empty abandoned allocations. A deterministic staging-remnant case verifies reclamation after interrupted manifest publication. It does not test simultaneous allocation starts.

Validation

On 0.67.8 (40e8207e), Linux x86-64, SBCL 2.6.6:

  • ./script/check --source passes.
  • The shell-log-resource and shell-log-job suites fail 1 of 13 cases on unchanged 40e8207e and pass 13/13 with this PR. The staging-remnant case alone fails on the first commit, 7240dd34.
  • Full ./script/check on 59ceb230: 1,234 cases, 2 failures. The same 2 fail on unchanged 40e8207e: test-telemetry-subprocess-deadlines (the saved-core child cannot load libcrypto.so.3) and test-model-catalog-standalone-http (bounded-body assertion).

@luciusmagn
luciusmagn force-pushed the fix/shell-log-cross-process-lifecycle branch from 59ceb23 to dd75787 Compare October 28, 2026 08:33
@luciusmagn
luciusmagn merged commit 7f5623d into lambda-symbolics:master Oct 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live shell logs appear interrupted or disappear across processes

2 participants