Map "job queue" to Standalone Activities + Add Ruby References - #258
Draft
starfleeth wants to merge 7 commits into
Draft
Map "job queue" to Standalone Activities + Add Ruby References#258starfleeth wants to merge 7 commits into
starfleeth wants to merge 7 commits into
Conversation
Prompts like "build a job queue with Temporal" or "does Temporal support a job queue?" had one incidental mention of the term to work from, so agents were free to answer with a Workflow-per-job, a Signal-driven dispatcher Workflow, or a description of Task Queues. This adds the explicit mapping onto Standalone Activities plus the code samples and SDK guides. - references/core/job-queue.md: new. Job-queue vocabulary mapped onto Temporal, which asks need a Workflow or Schedule instead, migration notes per source system (Celery/RQ, Sidekiq/Resque, BullMQ, SQS/Cloud Tasks, Hangfire), anti-patterns, code layout, and a per-language table of SDK guides and runnable samples. - SKILL.md: "background job queues" added to the trigger description, and a Job Queues section carrying the mapping and the note that Task Queues are the transport, not the queue producers enqueue into. - references/ruby/standalone-activities.md: new. Ruby supports Standalone Activities but had no reference file; written to match the other five. - references/core/standalone-activities.md: Ruby added to the supported SDK list, and a pointer to the new job-queue file. - references/core/patterns.md: job queue row in Choosing Between Patterns. Sample paths and docs URLs verified against the GitHub API and the docs repo. Conflict and reuse policy values verified against Temporal CLI 1.7.2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
starfleeth
force-pushed
the
feat/job-queue-standalone-activities
branch
from
July 31, 2026 21:54
da19301 to
9717fa0
Compare
starfleeth
changed the title
prasek
approved these changes
Aug 4, 2026
prasek
left a comment
prasek
left a comment
There was a problem hiding this comment.
Thanks @starfleeth overall lgtm! Dropped some comments/suggestions.
| - **No dead-letter queue to operate.** A job that exhausts its Retry Policy ends as a failed execution, retained in visibility with its last error and findable with a List Filter. | ||
| - **Retries, timeouts, and backoff are enforced by the platform**, not by a decorator argument the handler can ignore. | ||
| - **The same code graduates into orchestration.** One Activity Function runs as a background job today and as a step inside a multi-step Workflow tomorrow, with no code change and no Worker change. That upgrade path is the reason to pick Temporal over a job queue you would outgrow. | ||
| - **Cheaper than the usual workaround.** Wrapping a single Activity in a Workflow costs extra billable Actions in Temporal Cloud and extra Worker round-trips; a Standalone Activity avoids both. |
There was a problem hiding this comment.
Suggested change
| - **Cheaper than the usual workaround.** Wrapping a single Activity in a Workflow costs extra billable Actions in Temporal Cloud and extra Worker round-trips; a Standalone Activity avoids both. | |
| - **Cheaper than the usual workaround.** Wrapping a single Activity in a Workflow costs an extra billable Action in Temporal Cloud and extra Worker round-trips; a Standalone Activity avoids both. |
| | Worker process consuming the queue | Temporal Worker polling a Task Queue | | ||
| | Queue name / routing key | Task Queue name | | ||
| | Job ID | Activity ID — you choose it; use a business identifier | | ||
| | Result backend | The Activity's result, retrieved from the handle or `temporal activity result` | |
There was a problem hiding this comment.
Suggested change
| | Result backend | The Activity's result, retrieved from the handle or `temporal activity result` | | |
| | Result backend (a database or store you provide)| The Activity's result stored in the authoritative Activity record by Temporal Server, retrieved from the handle or `temporal activity result` | |
There was a problem hiding this comment.
Also note this is somewhat Celery specific but it's worthwhile to keep it
| | Long job keepalive / progress reporting | Activity Heartbeats, with heartbeat details for checkpointing | | ||
| | Cancel a job | `cancel` (cooperative, surfaced on the next heartbeat) or `terminate` (forceful) | | ||
| | Priority queues | Priority keys — free, Public Preview. See `references/core/priority-fairness.md` | | ||
| | Per-tenant fairness / avoiding noisy neighbors | Fairness keys and weights — Public Preview, and a paid feature in Temporal Cloud. See `references/core/priority-fairness.md` | |
There was a problem hiding this comment.
Suggested change
| | Per-tenant fairness / avoiding noisy neighbors | Fairness keys and weights — Public Preview, and a paid feature in Temporal Cloud. See `references/core/priority-fairness.md` | | |
| | Per-tenant fairness / avoiding noisy neighbors (usually not supported) | Fairness keys and weights — Public Preview, and a paid feature in Temporal Cloud. See `references/core/priority-fairness.md` | |
| | Cancel a job | `cancel` (cooperative, surfaced on the next heartbeat) or `terminate` (forceful) | | ||
| | Priority queues | Priority keys — free, Public Preview. See `references/core/priority-fairness.md` | | ||
| | Per-tenant fairness / avoiding noisy neighbors | Fairness keys and weights — Public Preview, and a paid feature in Temporal Cloud. See `references/core/priority-fairness.md` | | ||
| | Dashboard (Flower, Sidekiq Web, Bull Board) | Temporal Web UI, `temporal activity list` / `describe`, and the list/count client APIs | |
There was a problem hiding this comment.
Suggested change
| | Dashboard (Flower, Sidekiq Web, Bull Board) | Temporal Web UI, `temporal activity list` / `describe`, and the list/count client APIs | | |
| | Dashboard (Flower, Sidekiq Web, Bull Board) | Temporal Web UI, `temporal activity list` (with SearchAttribute support) / `describe`, and the list/count client APIs | |
| | Job metrics | Standard Activity metrics: scheduled, started, completed, failed, timed out, canceled | | ||
| | Manual/external job completion | Manual completion by Activity ID or task token | | ||
|
|
||
| On head-of-line blocking: a slow job occupies one Worker slot rather than stalling a single-threaded consumer, so one slow job does not block dispatch of the rest. Backlog-level starvation across tenants is a separate problem — by default Tasks dispatch FIFO, so a tenant enqueueing 100k jobs does put a small tenant behind the whole backlog. Fairness is what fixes that. |
There was a problem hiding this comment.
Suggested change
| On head-of-line blocking: a slow job occupies one Worker slot rather than stalling a single-threaded consumer, so one slow job does not block dispatch of the rest. Backlog-level starvation across tenants is a separate problem — by default Tasks dispatch FIFO, so a tenant enqueueing 100k jobs does put a small tenant behind the whole backlog. Fairness is what fixes that. | |
| On head-of-line blocking: a slow job occupies one Worker slot rather than stalling a single-threaded consumer, so one slow job does not block dispatch of the rest. Backlog-level starvation across tenants is a separate problem — by default Tasks dispatch FIFO, so a tenant enqueueing 100k jobs does put a small tenant behind the whole backlog. Fairness is what fixes that and in most cases only Temporal provides fairness. |
| 3. **`delay()` / `perform_async` / `queue.add()` → Client `start` or `execute`.** This is the only real call-site change. It happens in producer code, which must be non-Workflow application code. | ||
| 4. **Retry/timeout config → Retry Policy and Activity timeouts** on the start options, not on the handler. Remember the attempts-vs-retries off-by-one. | ||
| 5. **Job ID → Activity ID.** Reuse whatever idempotency key already exists. If there was none, derive one from the business entity. | ||
| 6. **Monitoring → visibility.** Replace Flower/Sidekiq Web/Bull Board polling of a Redis key with `list`/`count`/`describe` and the Web UI. |
There was a problem hiding this comment.
Suggested change
| 6. **Monitoring → visibility.** Replace Flower/Sidekiq Web/Bull Board polling of a Redis key with `list`/`count`/`describe` and the Web UI. | |
| 6. **Monitoring → built-in visibility and metrics.** Replace Flower/Sidekiq Web/Bull Board polling of a Redis key with `list`/`count`/`describe` and the Web UI. |
|
|
||
| Anti-patterns to avoid when building a job queue on Temporal: | ||
|
|
||
| 1. **A Workflow per job that runs exactly one Activity.** It costs extra billable Actions and extra Worker round-trips for no orchestration benefit. Prefer a Standalone Activity. (Exception: delayed and recurring jobs, per the table above.) |
There was a problem hiding this comment.
Suggested change
| 1. **A Workflow per job that runs exactly one Activity.** It costs extra billable Actions and extra Worker round-trips for no orchestration benefit. Prefer a Standalone Activity. (Exception: delayed and recurring jobs, per the table above.) | |
| 1. **A Workflow per job that runs exactly one Activity.** It costs an extra billable Action and extra Worker round-trips for no orchestration benefit. Prefer a Standalone Activity. (Exception: scheduled recurring jobs, per the table above. Note that non-recurring scheduled jobs via start delay is currently available) |
|
|
||
| - **The Activity Definition** — plain Activity code, identical to one written for a Workflow. | ||
| - **The Worker** — registers the Activity and polls the Task Queue. It does not know or care whether the Activity will be invoked standalone or from a Workflow. | ||
| - **The producer** — application code, an HTTP handler, or a CLI entry point that calls the Client. Never a Workflow. |
There was a problem hiding this comment.
Suggested change
| - **The producer** — application code, an HTTP handler, or a CLI entry point that calls the Client. Never a Workflow. | |
| - **The producer** — application code, an HTTP handler, or a CLI entry point that calls the Client. |
Note: There are some special cases where a Workflow wants to decouple the Activity lifetime from Workflow lifetime. Today a Workflow that wants to decouple the lifetime of the Activity from the Workflow must use a wrapper in-Workflow Activity to start a Standalone Activity using the SDK Client. In the future the Workflow will be able to start a Standalone Activity directly without a wrapper in-Workflow Activity.
| | Long-lived stateful entity | Entity Workflow | | ||
| | Safe retries/replays | Idempotency | | ||
| | Low-latency short operations | Local Activities | | ||
| | Run one background job (job queue) | Standalone Activity — see `references/core/job-queue.md` | |
There was a problem hiding this comment.
Suggested change
| | Run one background job (job queue) | Standalone Activity — see `references/core/job-queue.md` | | |
| | Run a background job (job queue) | Standalone Activity — see `references/core/job-queue.md` | |
|
|
||
| **Temporal's job queue is Standalone Activities.** When the developer asks for a job queue, background or async jobs, a work queue, or whether Temporal can replace Celery, Sidekiq, BullMQ, Resque, Hangfire, or SQS-plus-workers, build it with a Standalone Activity — not a Workflow wrapping a single Activity, and not a dispatcher Workflow that receives jobs by Signal. | ||
|
|
||
| Temporal **Task Queues** are the routing mechanism Workers poll, not a queue that producers push jobs into. Do not answer a job queue question by describing Task Queues. |
There was a problem hiding this comment.
Suggested change
| Temporal **Task Queues** are the routing mechanism Workers poll, not a queue that producers push jobs into. Do not answer a job queue question by describing Task Queues. | |
| Temporal **Task Queues** are the routing mechanism Workers poll, not a queue that producers push jobs into. Do not answer a job queue question by describing Temporal Task Queues. | |
| Note: users may ask "how to build a task queue with Temporal" or "can I use Temporal as a task queue" and in those cases (only in the user's request and not our reply) "task queue" may actually mean "job queue". For example Task nomenclature is used by Celery, Dramatiq, Huey and Asynq frameworks and users may refer to Celery as a "task queue" but this is a synonym for "job queue" which is the more prevalent term in industry. For example Job nomenclature is used by Hangfire, Sidekiq, Faktory, Bull, BullMQ, Resque, pg-boxx, Bee-Queue, RQ and users surveyed all recognized "job queue" as the most recognized term. |
- Delayed jobs take a start delay on the Standalone Activity itself, so they no longer route to a Workflow. Recurring jobs remain the exception, since Schedules start Workflows. - Note that a Workflow needing a job to outlive it starts the Standalone Activity from inside an in-Workflow Activity, so the producer is no longer described as "never a Workflow". - SKILL.md: "task queue" in a developer's request often means "job queue" (Celery, Dramatiq, Huey, Asynq use Task nomenclature; Sidekiq, Hangfire, BullMQ, Resque, RQ, Faktory use Job). - Wording per review: one billable Action rather than Actions, result backend as a store you provide, fairness usually unsupported elsewhere, Search Attribute support on activity list, built-in visibility and metrics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Thanks @prasek — all applied. Confirmed start delay on Standalone Activities via One question: |
- Unique-job keys need an ID conflict policy and an ID reuse policy; they answer different questions (currently running vs. already completed). Lists all accepted values, including AllowDuplicateFailedOnly, which is the closest match to "re-run only if the last attempt failed". - Narrow "only Temporal provides fairness" to the checkable claim that few job queues offer fairness keys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`start_delay` is a proto field name an agent cannot type at a shell; `--start-delay` on `temporal activity start` is the runnable equivalent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Neither claim was actionable for someone building a job queue: how many other job queues offer fairness keys, and whether Temporal is the right platform to pick, are not this file's job. The technical content stands without them — FIFO dispatch starves small tenants and Fairness fixes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- references/ruby/standalone-activities.md: the gem ships lib/temporalio/env_config.rb, so `require 'temporalio/envconfig'` raised LoadError in both snippets. samples-ruby uses env_config. - job-queue.md: start delay on Standalone Activities is implemented in chasm/lib/activity on server main, which carries v1.32.0-* dev tags, but no released Server exposes it (latest is v1.31.2). States the Server 1.32.0+ floor and drops the unqualified "any duration, at any scale". activity.startDelayEnabled defaults to true, so no flag flip is needed. - references/ruby/ruby.md: list standalone-activities.md in Reference Files, as the other five language index files do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
starfleeth
marked this pull request as draft
August 14, 2026 22:58
The 1.32.0 floor was inferred from an unreleased dev tag, and this PR is held until Standalone Activities go GA, by which point start delay is in the current release. Says to update to the latest Server instead of naming a version that cannot be verified while the PR sits in draft. A floor can go back once GA pins one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
"Build a job queue with Temporal" had one incidental mention of the term in this skill to work from, so agents were free to answer with a Workflow per job, a Signal-driven dispatcher Workflow, or an explanation of Task Queues. This maps job queue → Standalone Activities, with the samples and SDK guides.
Ref: Job Queue.
What
references/core/job-queue.md(new) — vocabulary mapping, which asks need a Workflow or Schedule instead, migration from Celery/RQ, Sidekiq/Resque, BullMQ, SQS/Cloud Tasks, Hangfire, anti-patterns, and a table of SDK guides + runnable samples for all six SDKs.SKILL.md— "background job queues" in the trigger description, plus a Job Queues section (including: Task Queues are the transport, not the job queue).references/ruby/standalone-activities.md(new) — Ruby supports Standalone Activities but had no reference file here.references/core/standalone-activities.md/patterns.md— Ruby added to supported SDKs, pointer to the new file, one row in Choosing Between Patterns.Notes
mdformat-clean.gotchas.md.start_delayon the start request,--start-delayontemporal activity start). Recurring jobs are the exception, since Schedules start Workflows — called out as such.🤖 Generated with Claude Code