[bug] Avoid holding the pipeline lock across management uart writes - #1206
Open
zeeshanlakhani wants to merge 1 commit into
Open
[bug] Avoid holding the pipeline lock across management uart writes#1206zeeshanlakhani wants to merge 1 commit into
zeeshanlakhani wants to merge 1 commit into
Conversation
zeeshanlakhani
force-pushed
the
zl/softnpu-mgmt-deadlock
branch
from
August 31, 2026 01:48
6c4faed to
3d694ff
Compare
zeeshanlakhani
force-pushed
the
zl/softnpu-mgmt-deadlock
branch
from
September 3, 2026 12:45
3d694ff to
b875003
Compare
`handle_management_message` locked the P4 pipeline before dispatching the request. For dump and radix requests, for example, it kept that lock while waiting for the guest to drain the 1-byte UART FIFO. In turn, if a queue notify message could send a vCPU into the `process_guest_packet` fn, where it waited on the same lock, that vCPU would not finish its PIO exit. This would lead to the guest not draining the UART, and, boom, deadlock. ## The Fix Now, the dump path copies table state while under the lock and writes it after unlocking happens. Table updates still hold the lock while changing the pipeline. UART writes now have a deadline, where if a response times out partway through, the handler will try to write the missing newline while waiting for the next request. *Note*: I ran into this issue on a long-running voxel session. The pre-fix binary reproduces it within minutes under a scadm polling loop, while the fixed one does not (A/B testing).
zeeshanlakhani
force-pushed
the
zl/softnpu-mgmt-deadlock
branch
from
September 5, 2026 15:31
b875003 to
c12da84
Compare
zeeshanlakhani
commented
Sep 5, 2026
Contributor
Author
There was a problem hiding this comment.
A clippy find on the falcon feature.
zeeshanlakhani
force-pushed
the
zl/softnpu-mgmt-deadlock
branch
from
September 5, 2026 15:32
c12da84 to
fb8e165
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
handle_management_messagelocked the P4 pipeline before dispatching the request. For dump and radix requests, for example, it kept that lock while waiting for the guest to drain the 1-byte UART FIFO.In turn, if a queue notify message could send a vCPU into the
process_guest_packetfn, where it waited on the same lock, that vCPU would not finish its PIO exit. This would lead to the guest not draining the UART, and, boom, deadlock.The Fix
Now, the dump path copies table state while under the lock and writes it after unlocking happens. Table updates still hold the lock while changing the pipeline. UART writes now have a deadline, where if a response times out partway through, the handler will try to write the missing newline while waiting for the next request.
Note: I ran into this issue on a long-running voxel session. The pre-fix binary reproduces it within minutes under a scadm polling loop, while the fixed one does not (A/B testing).