Repository navigation
Split a scheduled job into a secretless work phase and a credentialed publish phase - #197
Conversation
run.work (manifest) and work: true (chart) run the harness as a work init container under UID 10002 with no secret volume, its own bundle copy, and a shared job-stage emptyDir. The job container (UID 10001) starts only after every work process is dead, is the only one holding secrets and jobSecrets, reads the stage read-only via JOB_STAGE_DIR, and reports the work phase's exit as the job:<slug>:work gate. Part of sageox/sageox-monorepo#3777 Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
[High impact] The PR changes how opted-in scheduled jobs separate untrusted work from credentialed publishing. Fix the checkout, stage-path, and volume-conflict issues before merging.
|
Refuse a stage holding anything but directories and regular files before the publish body starts; open staged entries to the publisher's UID; withhold run.passthrough from the work phase; keep the work gate on an unstarted publish; resolve job run --stage before chdir; give the work HOME a safe.directory gitconfig for the read-only checkouts; refuse sharedVolumes named job-work/job-stage; fix the docs' secret ref example. Part of sageox/sageox-monorepo#3777 Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
On the CodeRabbit IRSA finding (exclude |
Part of sageox/sageox-monorepo#3777 Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An identity webhook (IRSA, EKS Pod Identity) injects into every container of the Pod, including the work init container, and its token file stays readable after the body's environment is scrubbed. The work phase now refuses to start its body when it finds an AWS credential variable or a well-known token file, and records the reason so the publish phase reports the work gate as UNKNOWN. Also moves the CHANGELOG entry back under Unreleased after the merge from main. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The publish container mounts job-stage at /stage; a sharedVolumes entry there gave the Pod a duplicate mount path, which Kubernetes rejects, so neither phase ran. Part of sageox/sageox-monorepo#3777 Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…work-phase Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Devin Review found 1 finding. Devin automatically fixed it.
Fixed by Devin (1)
- Shared stage mount blocks scheduled runs — Real: a sharedVolumes entry at /stage gave the publish container a duplicate mountPath. validate.yaml now refuses a shared mount at, inside, or containing /stage on an agent with a work: true job; jobs.sh covers it (red with the check reverted, green restored).
The fixtures copied process.env, so a shell holding an AWS credential variable, or a runner with a mounted token, made every work phase refuse before its body ran. JobHost takes an identityFileExists probe, and the fixtures strip the identity variables and stub the probe. Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…work-phase Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md
…ft there A work body that replaced work.json with a FIFO left the work phase blocked on a write-only open after its wall clock had stopped, so the publish phase never started. The record is written to a fresh file and renamed over the path. The missing-record test now gives the publish phase a sound stage, so its UNKNOWN can only come from the missing record. Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Summary
Part of sageox/sageox-monorepo#3777. On the Buzz job host the harness and the credential-holding runner share a UID and a container, so nothing the runner does can hold against code the harness started. This adds the approved split: run the harness in an init container with no secrets under a different UID, and publish from the main container, the only one with the secrets, handing the staged output across a shared
emptyDir.Opt-in. A job without it renders and runs exactly as before.
Manifest (
packages/core/src/manifest.ts):run.work: {command, args, wallClockMs}. Refused besidetrigger.onRequest(that run executes inside the gateway) andworker(single-container dispatcher Pod), and whenrun.work.wallClockMs + deadlineHeadroomMs >= budget.wallClockMs.Runtime (
packages/core/src/job-host.ts):JobHost.work()runsrun.work.commandwithsecrets/jobSecretsemptied, noJOB_VERDICT_PATH,JOB_PHASE=work, andJOB_STAGE_DIR=<stage>/out. It records{executed, exitCode}in<stage>/work.json.execute()with astageDirruns the publish body on what is left of the budget (publishPhase()), passes itJOB_STAGE_DIR, and adds ajob:<slug>:workgate read withO_NOFOLLOWand a size cap. A failed work phase means FAIL; a missing or garbled record means UNKNOWN.run.passthroughis the publish body's grant.CLI:
sageox-agent job work <slug> --stage <dir>refuses--secrets/--job-secrets/--paramand does not load.env.job runtakes--stage.Chart (
work: truein the job mirror):stage-work(init, UID 10002) makes the work phase its own bundle copy in ajob-workemptyDir.work(init, UID 10002) has nosecrets/jobSecretsvolume, noagent-data, checkouts read-only, and/stageread-write.job(UID 10001) is unchanged apart from getting--stage /stageand mounting the stage read-only.stage-workwrites/work/.gitconfigwithsafe.directory=*, so Git under UID 10002 accepts the read-only checkouts owned by 10001.sharedVolumesentry namedjob-work/job-stageor mounted at/over/stage(a duplicate mount path Kubernetes rejects), withserviceAccount.automountJobToken(a cluster token mounts into every container) and withpersistence.jobHome.Docs:
docs/job-contract.md("A job with a work phase") anddeploy/helm/README.md(same heading plus the container table). CHANGELOG entry added.Review focus
eks.amazonaws.com/skip-containersand says to check this on your own cluster. The work phase also fails closed at run time: it refuses to start its body when its container has an AWS credential variable or an IRSA, Pod Identity or cluster token file, and the work gate is then reported as UNKNOWN. How the chart keepsworkout of the injection is still an open question on the issue.work.jsonis the work phase's own claim, and its body can rewrite it. The docs say the publish body must verify what it publishes and must not trust a 0.Test Plan
deploy/helm/tests/jobs.sh(new block) asserts on rendered YAML:workcontainer runs as UID 10002 and thejobcontainer as 10001;workhas none ofsecrets-store,job-secrets-store,--secrets,--job-secrets,agent-data;jobkeeps its job secrets and mountsjob-stageread-only;work, andworkhas its ownstage-workbundle;templates/cronjob.yamlandtemplates/_helpers.tpland ran it:packages/core/test/job-phases.test.ts(16 tests, real node child processes):AWS_CONTAINER_CREDENTIALS_FULL_URIor a webhook token file, records the reason, and the publish run reports UNKNOWN. Proven red-first: with onlyjob-host.tsreverted, both tests fail (expected { executed: true, exitCode: 0 });GH_TOKEN,GH_APP_PEMorJOB_VERDICT_PATH, while the publish body gets both secrets and reads what work staged;run.passthroughis withheld from work, and 0700/0600 staged entries are opened to other UIDs.pnpm typecheckpasses. All fivedeploy/helm/tests/*.shpass.helm lint --strictpasses with and withoutwork: true(Helm 4.2.0).pnpm testhad failures inteam-server.test.ts,agent-cwd.test.tsandvault.test.ts: ledger and ox-binary interaction, plusagenot being installed at first. None of those files touch this diff. CI is the gate.Leftovers (not exercised in CI)
jobstarts only afterwork, including aftersetsid()descendants have exited;fsGrouplets UID 10002 write/stageand UID 10001 read it;envinsideworkhas noAWS_*credential variables and no Pod Identity endpoint under the reference EKS composition.run.workand validate the staged output in the publish body. That work lives in sageox-monorepo.workandstage-work, regardless ofautomountServiceAccountToken. The work phase now fails closed at run time (refuses to start, work gate UNKNOWN), so a phased job under a Pod-wide identity does not run until the chart keepsworkout of the injection — that chart-side choice is still open on https://github.com/sageox/sageox-monorepo/issues/3777#issuecomment-6864020041.Checklist (expand if needed)
pnpm typecheck && pnpm testgreen (typecheck green; test suite — see note above, CI is the gate)deploy/changedLink to Devin session: https://app.devin.ai/sessions/75d88b208d7046a6afa395bead550465
Open in Devin Desktop: https://app.devin.ai/desktop/session/75d88b208d7046a6afa395bead550465?variant=devin
🟢 Devin Review found nothing to fix · View findings · Follow Devin's work
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Summary by CodeRabbit