Skip to content

Flaky Rust test: notification_poll_failures_do_not_stop_the_client hits database is locked #1455

Description

@bgentry

@brandur notification_poll_failures_do_not_stop_the_client in rust/riverqueue/tests/resilience_sqlite.rs (added with the Rust port in #1442) fails intermittently in CI with SQLite's database is locked:

thread 'notification_poll_failures_do_not_stop_the_client' panicked at riverqueue/tests/resilience_sqlite.rs:208:10:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 5, message: "database is locked" })

Line 208 is the unwrap in the job_state helper, which the test calls from wait_until while waiting for the job to complete.

It's shown up in about 6 of the ~12 Rust workflow runs since #1442 merged, 8 jobs in all, always with the same panic. It doesn't follow a particular PostgreSQL or Rust version:

The likely cause is the test's own setup. It opens its pool in rollback-journal (Delete) mode with a 20 ms busy timeout, so that the outbox poll fails quickly while a second connection holds BEGIN EXCLUSIVE. The client and the test's job_state reads share that pool. After the exclusive lock is released, the client keeps writing (fetch polls every 20 ms with a 1 ms cooldown, plus the insert and completion), and in rollback-journal mode a committing writer blocks readers. On a loaded runner a commit can take longer than 20 ms, so job_state gets SQLITE_BUSY and its unwrap panics. The client.insert just before the wait uses the same pool and has the same exposure.

Two possible fixes: read job state for the assertions through a separate pool with a normal busy timeout, or have the wait_until condition treat a busy error as "not yet" and keep polling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions