Repository navigation
Fix QUIC UDP recv busy-loop and recover dead stacks after backgrounding - #205
panfilov-vladislav wants to merge 2 commits into
Conversation
poll_recv() only special-cased WouldBlock from try_io(); any other error (e.g. the network interface going down, which happens when iOS suspends networking for a backgrounded app) was silently discarded and the loop went straight back to poll_recv_ready(). Since try_io() only clears the reactor's readiness bit for WouldBlock, a real error leaves the socket marked ready, so the next iteration immediately re-entered recv(), hit the same error, and repeated - a synchronous spin inside a single poll() call that pins a tokio worker thread at 100% CPU instead of ever yielding back to the executor. Propagate non-WouldBlock errors instead so the caller can handle or recreate the connection.
poll_recv() now correctly surfaces a fatal UDP error instead of busy-looping (previous commit), but quinn's endpoint driver treats that as terminal and nothing ever recreates the socket - the whole QUIC stack (and with it, DHT/ peer connectivity) stays dead until the app restarts, even though the actual network interface recovers (e.g. once the app leaves the background). Inner::bind() only rebinds a stack when its address differs from before, which never happens here (backgrounding doesn't change the IP/port, just kills the socket's liveness), so a stack that died was never getting recreated even when something called bind() again with the same addresses. Track a `died` flag on CustomUdpSocket, set it when poll_recv hits a fatal (non-WouldBlock) error, and surface it through Connector/QuicStack/Gateway so Inner::bind() also rebinds when any current stack is dead, regardless of whether the address changed.
Found potential problems after I approved.
madadam
left a comment
There was a problem hiding this comment.
I think there is a potential race condition here. Are we sure the is_dead flag is always going to be set before we try to rebind the network? The way it's implemented in the PR might still work in practice but only accidentally due to timing: we are continuously receiving on the socket so is_dead is likely to get set before the app gets notified about being resumed, but I'm not sure we can 100% rely on this.
One (incomplete) idea how to fix this: instead of tracking the is_dead flag, we can "probe" the socket before trying to rebind it. If the probe tells us the socket is dead, we rebind, otherwise we don't. I'm not sure how to implement such "probe" though...
|
I believe this is a problem Previously we just ignored every error, which for the background use case was a problem, now we allow any error except |
|
Could you do the "is stack dead" check inside the This is a smaller one, but if you're going to do more changes to the code it would be nice to include it. |
|
There might be a better way to do all this: The whole logic of detecting socket errors and rebinding it can be fully encapsulated in the |
Problem
When the OS tears down networking for a backgrounded app (e.g. iOS suspending
the network interface), the QUIC UDP socket starts returning fatal recv errors.
Two separate bugs made this unrecoverable:
100% CPU busy-loop.
CustomUdpSocket::poll_recv()only special-casedWouldBlockfromtry_io(); any other error was silently discarded and theloop went straight back to
poll_recv_ready(). Becausetry_io()onlyclears the reactor's readiness bit for
WouldBlock, a real error left thesocket marked ready, so the next iteration immediately re-entered
recv(),hit the same error, and repeated — a synchronous spin inside a single
poll()that pins a tokio worker at 100% CPU instead of ever yielding.Stack never recovers. Even once recv errors surface correctly, quinn's
endpoint driver treats a fatal socket error as terminal and nothing
recreates the socket.
Inner::bind()only rebinds a stack when its addressdiffers, but backgrounding kills socket liveness without changing the IP/port
— so the dead stack (and with it DHT/peer connectivity) stayed dead until the
app restarted, even after the network interface recovered.
Fix
net/src/quic.rs: propagate non-WouldBlockrecv errors instead ofdiscarding them, and track a
diedflag onCustomUdpSocket, exposed viaConnector::is_dead().lib/src/network/: surface that flag up throughQuicStack→Stacks→
Gateway::any_stack_is_dead(), and makeInner::bind()force a rebind whenany current stack is dead, regardless of whether the bind address changed.