finagle a STREAMS rx buffer size for DLPI handles - #1204
Conversation
|
|
||
| let mut offset = 1; | ||
| for de in &dir[msg.offset as usize..] { | ||
| for (offset, de) in (1..).zip(dir[msg.offset as usize..].iter()) { |
There was a problem hiding this comment.
Thsi is addressing a recent clippy complaint.
iximeow
left a comment
iximeow
left a comment
There was a problem hiding this comment.
abstractly i Get How It Could Work and i trust you to have tested that softnpu works better with this change, but i'll admit i don't understand the mechanics of why These Flags do the needful..
anyhow, as you might have seen in the other PRs here, the header-check failure is an unfortunate consequence of when the viona API changes got into stlouis vs the Propolis side not being in yet, so feel free to merge even though that check's failing.
| // Disable chunking, still keeps the water mark that was applied for the | ||
| // supplied chunk. Yes this is relying on implicit behavior. Good motivation | ||
| // to get off DLPI entirely. |
There was a problem hiding this comment.
I figured I dunno how this works and it'd be interesting to fish through, my best guess is that doing all this to push q_hiwat to be large enough to at least contain this chunk since we've set it SB_NO_DROPS preventing the kernel from just dropping it? in which case, presumably we can't somehow strqset(.., QHIWAT, idk?, 0x120000)?
There was a problem hiding this comment.
Pushing the bufmod streams module is purely for the side-effect of impacting the high water mark. I set SB_NO_DROPS based on the comment in uts/common/io/bufmod.c that says
SB_NO_DROPS - bufmod behaves transparently in flow control and propagates the blocked flow
conditions downstream.
With the idea being having the bufmod be a pure passthrough. Afaict strqset is only available in the kernel so this is the best I could come up with. The snoop program does the same trick to effectively get a decent size receive buffer.
There was a problem hiding this comment.
oh, as i was looking through kernel source and strqset looked vaguely like how i've seen ioctls get plumbed through, but i see now that it's under Kernel Functions for Drivers so that explains the interface and why it wouldn't be usable here. welp. thanks for the pointers 🫡
Yep, with this change I'm able to push ~1.5 gbps through a softnpu port. Up from ~5 mbps. |
3 participants
STREAMS uses a high water mark as a back pressure mechanism. When we hit the high water mark, messages are dropped until we drain to the low water mark. This essentially makes the high water mark a receive buffer size.
For TCP on the STREAMS path (i.e. TPI consumers), it appears that the high water mark is set to SO_RCVBUF which is 128000 bytes. However, for DLPI the high water mark is not set and it defaults to 5120. This meaans we hit the mark at the first jumbo frame and thrash from there.
It seems the only way to influence this outside the kernel is pushing a passthrough bufmod STREAMS module. So that's what we do here for the time being.
It looks like the CI failure is addressed by #1093