Skip to content

test(vault): separate the client timeout from sqlx acquire - #370

Merged
LKSNDRTMLKV merged 1 commit into
mainfrom
fix/publish-serve-cycle-does-not-hang
Sep 17, 2026
Merged

LKSNDRTMLKV merged 1 commit into
mainfrom
fix/publish-serve-cycle-does-not-hang

Conversation

@LKSNDRTMLKV

Copy link
Copy Markdown
Member

Relates to #336. Does not close it — this is a narrowing, not a fix, and it should not be merged as though the flake were solved.

What I could not do

Reproduce it. Eight consecutive runs of publish_serve_cycle against Docker, all green, ~4.7 s each against a 10-second slow-test budget and a 120-second harness ceiling. That matches the issue's own observation that a re-run on the same commit passed, and it means the stall needs CI load to appear.

No root cause, so no fix. A guess at a flake is worse than none, because it looks solved and the next occurrence gets attributed to something else.

What the investigation did establish

The instrumentation this issue asks for already exists, and its author said exactly what it was for. #340 (6796e87) gave TestClient a request timeout and named-URL reporting, closing with "This does not fix a stall. It makes the next one diagnosable, which is the prerequisite for fixing it."

So the issue's title is stale: post-#340 a stall does not reach the 120-second ceiling, it fails at thirty with a named URL.

And it has now paid out. The first real data point this issue has ever had arrived today, on an unrelated dependency bump:

GET http://127.0.0.1:33981/public/dpp/01a0aedc-… failed
  (timed out — the server accepted the request and never answered)

Two things follow that were not known before: it is the public read that stalls, not the publish, and the server had already accepted the connection — so this is not startup, port binding or container readiness.

🚨 The flaw this change fixes

PgDal::connect builds its pool with max_connections(10) and no explicit acquire_timeout, so sqlx applies its default — which is also thirty seconds.

The two candidate explanations for "accepted and never answered" are:

  1. the pool had no connection to give, and the handler is parked in acquire;
  2. the handler is stuck on something else.

At thirty-all they expire together. Whichever fires first is a race, and the failure line is identical either way — so every occurrence costs a CI cycle and distinguishes nothing. That is the outcome #340 existed to prevent, defeated by a coincidence of defaults.

Raising the client to 45 s separates them. If the cause is pool starvation, the request now returns a 500 naming the acquire failure while this client is still waiting. If it is anything else, the named-URL timeout fires exactly as before. Still far below the 120-second ceiling, so a stall remains a failed request rather than a killed test.

One observation will now say which of the two it is.

Why the timeout moved rather than the pool

Setting an explicit, shorter acquire_timeout on PgDal::connect would separate them too, and more directly. It is not done here because that is production wiring, and changing how long a live node waits for a database connection to make a test suite easier to diagnose is the wrong reason to touch it. If a shorter acquire timeout is right, it is right on its own merits and belongs in its own change.

This is one constant in a test helper.

just check green.

@LKSNDRTMLKV
LKSNDRTMLKV force-pushed the fix/publish-serve-cycle-does-not-hang branch from 0015164 to f331730 Compare September 17, 2026 13:57
@LKSNDRTMLKV
LKSNDRTMLKV merged commit 1b73c19 into main Sep 17, 2026
14 checks passed
@LKSNDRTMLKV
LKSNDRTMLKV deleted the fix/publish-serve-cycle-does-not-hang branch September 17, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant