diff --git a/README.md b/README.md index cc8a4760..67d6a6f5 100644 --- a/README.md +++ b/README.md @@ -341,6 +341,41 @@ Were a consumer to _raise an exception_ or _outright crash_ or _have network pro In the event of a consumer crash or (ephemeral) network problems, we do not want work to get lost. The opsqueue system takes the 'at least once' approach (rather than the 'at most once' approach). This means that your consumers **must be idempotent**. They have to handle the possibility of (part of a) chunk being re-executed multiple times. +## Chunks + +It may be helpful in understanding OpsQueue to see a state machine of a chunk. +Definitions of each of the states follows below: + +```mermaid +stateDiagram + [*] --> Available: Submission created + [*] --> Paused: Submission created paused + Paused --> Available: Submission unpaused + Available --> Reserved: Consumer reserves chunk + Reserved --> Completed: Consumer reports success, completion recorded + Reserved --> Failed: Consumer reports failure, retry limit reached + Reserved --> Available: Consumer reports failure, retries remain + Reserved --> Available: Consumer disconnects, reservation released + Reserved --> Available: Reservation expires + Reserved --> Available: Failure transaction fails, reservation released + Paused --> Skipped: Submission cancelled + Available --> Skipped: Submission cancelled or failed + Reserved --> Skipped: Submission cancelled or failed +``` + +Note that late (after reservation expired) completion/failure of a chunk is also +possible, so we would have additional transitions `Available -> Completed` and +`Available -> Failed`, but kept out of the diagram for simplicity. + +| State | Pseudo-SQL definition | +|---------------|---------------------------------------------------------------------------------------| +| **Paused** | `SELECT * FROM chunks_paused` | +| **Available** | `SELECT * FROM chunks WHERE opsqueue_is_reserved(submission_id, chunk_index) = FALSE` | +| **Reserved** | `SELECT * FROM chunks WHERE opsqueue_is_reserved(submission_id, chunk_index) = TRUE` | +| **Completed** | `SELECT * FROM chunks_completed` | +| **Failed** | `SELECT * FROM chunks_failed WHERE skipped = FALSE` | +| **Skipped** | `SELECT * FROM chunks_failed WHERE skipped = TRUE` | + ## API connections Under the hood, the producer and the queue talk with each other using a JSON-REST API over HTTP. Users of opsqueue don't need to think about this, as this is abstracted behind the client library. diff --git a/opsqueue/src/consumer/server/mod.rs b/opsqueue/src/consumer/server/mod.rs index 5cd9a224..18855f85 100644 --- a/opsqueue/src/consumer/server/mod.rs +++ b/opsqueue/src/consumer/server/mod.rs @@ -235,12 +235,11 @@ impl Completer { output_content, reservations, } => { - // Even in the unlikely event that the DB write fails, - // we still want to unreserve the chunk let db_res = crate::common::chunk::db::complete_chunk(id, output_content, &mut conn) .await; - + // Even in the unlikely event that the DB write fails, + // we still want to unreserve the chunk reservations.lock().expect("No poison").remove(&id); if let Some(started_at) = self .dispatcher @@ -267,8 +266,6 @@ impl Completer { failure, reservations, } => { - // Even in the unlikely event that the DB write fails, - // we still want to unreserve the chunk let failed_permanently = crate::common::chunk::db::retry_or_fail_chunk( id, failure, @@ -276,6 +273,8 @@ impl Completer { self.max_chunk_retries, ) .await; + // Even in the unlikely event that the DB write fails, + // we still want to unreserve the chunk reservations.lock().expect("No poison").remove(&id); let maybe_started_at = self .dispatcher