Skip to content

Split a scheduled job into a secretless work phase and a credentialed publish phase - #197

Merged
devin-ai-integration[bot] merged 10 commits into
mainfrom
devin/1794600922-job-work-phase
Nov 19, 2026
Merged

devin-ai-integration[bot] merged 10 commits into
mainfrom
devin/1794600922-job-work-phase

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part of sageox/sageox-monorepo#3777. On the Buzz job host the harness and the credential-holding runner share a UID and a container, so nothing the runner does can hold against code the harness started. This adds the approved split: run the harness in an init container with no secrets under a different UID, and publish from the main container, the only one with the secrets, handing the staged output across a shared emptyDir.

Opt-in. A job without it renders and runs exactly as before.

Manifest (packages/core/src/manifest.ts): run.work: {command, args, wallClockMs}. Refused beside trigger.onRequest (that run executes inside the gateway) and worker (single-container dispatcher Pod), and when run.work.wallClockMs + deadlineHeadroomMs >= budget.wallClockMs.

Runtime (packages/core/src/job-host.ts):

  • JobHost.work() runs run.work.command with secrets/jobSecrets emptied, no JOB_VERDICT_PATH, JOB_PHASE=work, and JOB_STAGE_DIR=<stage>/out. It records {executed, exitCode} in <stage>/work.json.
  • execute() with a stageDir runs the publish body on what is left of the budget (publishPhase()), passes it JOB_STAGE_DIR, and adds a job:<slug>:work gate read with O_NOFOLLOW and a size cap. A failed work phase means FAIL; a missing or garbled record means UNKNOWN.
  • A phased job with no stage, or an unphased job with one, is refused.
  • Before the publish body starts, it refuses a stage that holds anything other than directories and regular files, and caps the number of entries. A staged symlink would resolve inside the publish container, which holds the secrets.
  • After the work body exits, the work phase makes what it staged readable to the publisher's UID. It gets only the base passthrough list; run.passthrough is the publish body's grant.
  • The work gate is kept when the publish body never starts, so a failed work phase still reports FAIL.

CLI: sageox-agent job work <slug> --stage <dir> refuses --secrets/--job-secrets/--param and does not load .env. job run takes --stage.

Chart (work: true in the job mirror):

  • stage-work (init, UID 10002) makes the work phase its own bundle copy in a job-work emptyDir.
  • work (init, UID 10002) has no secrets/jobSecrets volume, no agent-data, checkouts read-only, and /stage read-write.
  • job (UID 10001) is unchanged apart from getting --stage /stage and mounting the stage read-only.
  • stage-work writes /work/.gitconfig with safe.directory=*, so Git under UID 10002 accepts the read-only checkouts owned by 10001.
  • Refused with a sharedVolumes entry named job-work/job-stage or mounted at/over /stage (a duplicate mount path Kubernetes rejects), with serviceAccount.automountJobToken (a cluster token mounts into every container) and with persistence.jobHome.

Docs: docs/job-contract.md ("A job with a work phase") and deploy/helm/README.md (same heading plus the container table). CHANGELOG entry added.

Review focus

  • Pod-wide cloud identity is not withheld by this chart. The IRSA webhook and EKS Pod Identity inject into every container unless they are told otherwise. The README covers eks.amazonaws.com/skip-containers and says to check this on your own cluster. The work phase also fails closed at run time: it refuses to start its body when its container has an AWS credential variable or an IRSA, Pod Identity or cluster token file, and the work gate is then reported as UNKNOWN. How the chart keeps work out of the injection is still an open question on the issue.
  • work.json is the work phase's own claim, and its body can rewrite it. The docs say the publish body must verify what it publishes and must not trust a 0.

Test Plan

  • deploy/helm/tests/jobs.sh (new block) asserts on rendered YAML:
    • the work container runs as UID 10002 and the job container as 10001;
    • work has none of secrets-store, job-secrets-store, --secrets, --job-secrets, agent-data;
    • job keeps its job secrets and mounts job-stage read-only;
    • checkouts are read-only in work, and work has its own stage-work bundle;
    • none of this appears on a job without the flag;
    • the worker, prompt-job and cluster-token combinations are refused.
  • Red-first. I reverted only templates/cronjob.yaml and templates/_helpers.tpl and ran it:
    --- fix reverted (cronjob.yaml, _helpers.tpl) ---
    jobs.sh: a job with work: true renders no work init container
    exit=1
    --- fix restored ---
    jobs.sh: ok
    exit=0
    
  • packages/core/test/job-phases.test.ts (16 tests, real node child processes):
    • the work phase refuses to start beside an injected AWS_CONTAINER_CREDENTIALS_FULL_URI or a webhook token file, records the reason, and the publish run reports UNKNOWN. Proven red-first: with only job-host.ts reverted, both tests fail (expected { executed: true, exitCode: 0 });
    • the work body's env has no GH_TOKEN, GH_APP_PEM or JOB_VERDICT_PATH, while the publish body gets both secrets and reads what work staged;
    • FAIL when work exits non-zero, UNKNOWN when there is no record, and the stage/phase mismatch is refused;
    • the manifest refusals and the budget arithmetic;
    • a symlinked stage entry is refused before the publish body runs, a failed work phase stays FAIL when publish can't start, run.passthrough is withheld from work, and 0700/0600 staged entries are opened to other UIDs.
  • pnpm typecheck passes. All five deploy/helm/tests/*.sh pass. helm lint --strict passes with and without work: true (Helm 4.2.0).
  • Locally, pnpm test had failures in team-server.test.ts, agent-cwd.test.ts and vault.test.ts: ledger and ox-binary interaction, plus age not being installed at first. None of those files touch this diff. CI is the gate.

Leftovers (not exercised in CI)

  • Cluster check. On a real cluster, confirm that:
    • job starts only after work, including after setsid() descendants have exited;
    • fsGroup lets UID 10002 write /stage and UID 10001 read it;
    • env inside work has no AWS_* credential variables and no Pod Identity endpoint under the reference EKS composition.
  • Consumer migration. The monorepo's Buzz jobs (Whittle and others) still need to move their harness into run.work and validate the staged output in the publish body. That work lives in sageox-monorepo.
  • Cloud identity (chart side). Under EKS Pod Identity or IRSA, AWS mutates every container in the Pod, including work and stage-work, regardless of automountServiceAccountToken. The work phase now fails closed at run time (refuses to start, work gate UNKNOWN), so a phased job under a Pod-wide identity does not run until the chart keeps work out of the injection — that chart-side choice is still open on https://github.com/sageox/sageox-monorepo/issues/3777#issuecomment-6864020041.

Checklist (expand if needed)
  • Tests added, or N/A documented
  • pnpm typecheck && pnpm test green (typecheck green; test suite — see note above, CI is the gate)
  • Deployment checks run if deploy/ changed
  • CHANGELOG entry added if user-facing
  • Breaking change documented (none: opt-in)
  • Docs updated if behavior or configuration changed

Link to Devin session: https://app.devin.ai/sessions/75d88b208d7046a6afa395bead550465
Open in Devin Desktop: https://app.devin.ai/desktop/session/75d88b208d7046a6afa395bead550465?variant=devin


🟢 Devin Review found nothing to fix · View findings · Follow Devin's work

Devin Review


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • New Features
    • Scheduled process jobs can now run an optional work phase before publishing results. Work runs separately from the credentialed publish phase, with its own time limit and isolated access.
    • Publish jobs can read staged work output. Results include a check for whether the work phase succeeded.
    • The command-line interface and Helm chart support configuring and running work phases.
  • Validation & Safety
    • Work phases are rejected for unsupported job configurations or conflicting shared volumes.
    • Work will not start when recognized cloud credentials or identity tokens are present in its container. Staged output is checked before publishing.
  • Documentation
    • Added guidance on configuring work phases, their limits, and publish behavior.

run.work (manifest) and work: true (chart) run the harness as a work init container under UID 10002 with no secret volume, its own bundle copy, and a shared job-stage emptyDir. The job container (UID 10001) starts only after every work process is dead, is the only one holding secrets and jobSecrets, reads the stage read-only via JOB_STAGE_DIR, and reports the work phase's exit as the job:<slug>:work gate.

Part of sageox/sageox-monorepo#3777

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@veria-ai

veria-ai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Fixed/addressed: 3 · PR risk: 0/10

Reviews

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review
Walkthrough

Scheduled process jobs can now run an optional work phase before publishing. The work command runs without resolved job secrets or a verdict path and writes staged output. Publishing validates the stage, uses a reduced time budget, and includes the work result in its verdict.

Changes: Scheduled job work phases

Layer / File(s) Summary
Work phase contract and execution
packages/core/src/manifest.ts, packages/core/src/job-host.ts, packages/core/test/job-phases.test.ts, docs/job-contract.md, CHANGELOG.md
The manifest accepts run.work with a command, arguments, and wall-clock limit. JobHost runs the work command without resolved secrets or a verdict path, records its outcome, validates staged output, and includes the work result in the publish verdict. Tests cover validation, budgets, staged output, and missing or failed work records.
CLI work command
packages/cli/src/args.ts, packages/cli/src/cli.ts
The CLI adds job work <slug> --stage <dir>, validates its inputs, and passes the stage directory to work or publish execution.
Helm work-pod orchestration
deploy/helm/values.schema.json, deploy/helm/templates/_helpers.tpl, deploy/helm/templates/cronjob.yaml, deploy/helm/templates/validate.yaml, deploy/helm/tests/jobs.sh, deploy/helm/README.md
The chart adds work-phase containers and staging volumes for work: true jobs. It rejects incompatible configurations and adds rendered-output checks and usage documentation.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant JobHost
  participant StageDirectory
  CLI->>JobHost: invoke job work with stage directory
  JobHost->>StageDirectory: write staged output and work record
  CLI->>JobHost: invoke job run with stage directory
  JobHost->>StageDirectory: read work record and validate staged output
  JobHost->>JobHost: run publish phase and include work check
Loading

Suggested reviewers: shrimalmadhur

Merge Risk: Low · previous review def30

Phased jobs are opt-in, so existing jobs are unaffected. A work body can leave its own job's init container blocked until the Pod deadline by replacing the work record with a FIFO. The missing-record test also needs a valid stage directory so it protects the behavior it is named for. Both are small fixes, so this is mergeable with low risk once they are addressed or accepted.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title states the main change: a scheduled job is split into a work phase that runs without secrets and a publish phase that runs with credentials. The raw summary and objectives describe this same…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title states the main change: a scheduled job is split into a work phase that runs without secrets and a publish phase that runs with credentials. The raw summary and objectives describe this same two-phase flow, including the manifest field, runtime gates, CLI command, and Helm chart changes. The title is specific and concise, and a teammate scanning history would understand the primary change.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

veria-ai[bot]

This comment was marked as resolved.

@greptile-apps

greptile-apps Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 2/5 Tier: plus

[High impact] The PR changes how opted-in scheduled jobs separate untrusted work from credentialed publishing.

Fix the checkout, stage-path, and volume-conflict issues before merging.

Fix All in Claude CodeFindings

  1. P1 Git rejects shared checkouts ▶
  2. P1 Relative stage paths point elsewhere ▶
  3. P1 Shared volumes break phased Pods ▶
  4. P2 Publish crashes hide work failures ▶

Diagram

sequenceDiagram
    participant W as Work init container (UID 10002)
    participant S as Shared stage
    participant P as Publish container (UID 10001)
    participant C as Secret mounts
    W->>S: Write output and work.json
    Note over W,P: Work container exits before publish starts
    C->>P: Provide publish credentials
    P->>S: Publish command reads staged output
    P->>S: Host reads work.json
    P->>P: Combine work and publish checks
Loading

Reviews (1) · Last reviewed commit: "Split a scheduled job into a secretless ..." · Reviewed by Greptile

greptile-apps[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

Refuse a stage holding anything but directories and regular files before the publish body starts; open staged entries to the publisher's UID; withhold run.passthrough from the work phase; keep the work gate on an unstarted publish; resolve job run --stage before chdir; give the work HOME a safe.directory gitconfig for the read-only checkouts; refuse sharedVolumes named job-work/job-stage; fix the docs' secret ref example.

Part of sageox/sageox-monorepo#3777

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

On the CodeRabbit IRSA finding (exclude work/stage-work from the publish cloud identity): whether the chart should set eks.amazonaws.com/skip-containers itself, leave it to the operator, or route phased jobs' secrets some other way is a product decision not covered by the approved recommendation. It's raised for Ryan at https://github.com/sageox/sageox-monorepo/issues/3777#issuecomment-6864020041. Until then the README documents it, and verifying it on the cluster is a PR leftover.

veria-ai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

rsnodgrass and others added 5 commits November 13, 2026 20:54
Part of sageox/sageox-monorepo#3777

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An identity webhook (IRSA, EKS Pod Identity) injects into every container of the Pod, including the work init container, and its token file stays readable after the body's environment is scrubbed. The work phase now refuses to start its body when it finds an AWS credential variable or a well-known token file, and records the reason so the publish phase reports the work gate as UNKNOWN. Also moves the CHANGELOG entry back under Unreleased after the merge from main.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The publish container mounts job-stage at /stage; a sharedVolumes entry there gave the Pod a duplicate mount path, which Kubernetes rejects, so neither phase ran.

Part of sageox/sageox-monorepo#3777

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…work-phase

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 finding. Devin automatically fixed it.

Fixed by Devin (1)
  • Shared stage mount blocks scheduled runs — Real: a sharedVolumes entry at /stage gave the publish container a duplicate mountPath. validate.yaml now refuses a shared mount at, inside, or containing /stage on an agent with a work: true job; jobs.sh covers it (red with the check reverted, green restored).

View findings in Devin Review · Follow Devin's work

Devin Review

coderabbitai[bot]

This comment was marked as resolved.

The fixtures copied process.env, so a shell holding an AWS credential variable, or a runner with a mounted token, made every work phase refuse before its body ran. JobHost takes an identityFileExists probe, and the fixtures strip the identity variables and stub the probe.

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
coderabbitai[bot]

This comment was marked as resolved.

rsnodgrass and others added 2 commits November 19, 2026 17:33
…work-phase

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	CHANGELOG.md
…ft there

A work body that replaced work.json with a FIFO left the work phase blocked on a write-only open after its wall clock had stopped, so the publish phase never started. The record is written to a fresh file and renamed over the path. The missing-record test now gives the publish phase a sound stage, so its UNKNOWN can only come from the missing record.

Co-Authored-By: Ryan Snodgrass <ryan@sageox.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration
devin-ai-integration Bot merged commit 7a474db into main Nov 19, 2026
7 checks passed
@devin-ai-integration
devin-ai-integration Bot deleted the devin/1794600922-job-work-phase branch November 19, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant