[Storage] Optimize IOPS for RAID-0 NVMe disks

[Badrish Chandramouli (badrishc)]

Description of Change

Garnet's Native storage device (the C++ shim over Linux libaio / io_uring, and IOCP on
Windows) could not keep a fast NVMe array busy: on an 8×NVMe RAID-0 whose fio 4K random-read
ceiling is 8.24 M IOPS, the device topped out around 4.1 M and disk-served RESP GET
throughput was correspondingly capped. This PR removes the submit/completion bottlenecks so the
device reaches fio parity and the Tsavorite KV layer follows.

Results on the reference host (8×NVMe RAID-0, 100% random reads served from disk):

layer	before	after	vs fio ceiling
raw device (Device.benchmark, libaio)	4.14 M	8.09–8.23 M	~100%
raw device (Device.benchmark, io_uring)	—	8.41–8.45 M	~100%
Tsavorite KV (KV.benchmark, disk-bound)	—	6.3 M	~77%

The KV gap is Tsavorite managed per-op CPU (hash lookup, pending context, completion dispatch),
not the device. The three benchmarks use different datasets and configurations, so their absolute
numbers are not directly comparable.

Key changes:

• Per-submitter sharding of in-flight tracking. A single global pending counter plus a shared
  free-slot queue were the dominant cost at high IOPS. In-flight is now tracked per submitter
  thread across NumShards cache-line-padded shards, each with its own completion-slot free list.
  activeShards is maintained exactly on the 0→1 / 1→0 transitions so the per-thread throttle
  divisor tracks live submitters with no global counter and no reconciliation.
• Ring count decoupled from drainer count. io_uring is ring-starved below submitter
  concurrency; previously rings were tied to the completion-thread count. --device-io-contexts
  now sets the ring count independently, and its smart default
  (max(completion-threads, min(2 × ProcessorCount, 64))) sizes rings to the hardware with no user action.
• Affine inline drain. TryCompleteMine() drains only the calling thread's own ring instead of
  walking every ring, cutting syscalls N× and removing cross-context ring-lock contention.
  AllocatorBase.AsyncGetFromDisk's throttle-wait uses it.
• Multiple completion drain threads (--device-completion-threads, default 4). A single
  drainer convoys on the completion-signal path and collapses throughput under load.
• libaio reservation sized against the global fs.aio-max-nr budget instead of blindly
  requesting a deep ring, so many devices (e.g. cluster aux logs) still start on a stock host.
  The per-device budget ceiling runs last and caps aggregate in-flight along with it, so a stock
  65536 budget halves the default 4096 throttle (~9% on libaio disk serving). A disk-serving host
  should size fs.aio-max-nr / --device-aio-max-devices ≥ --device-throttle-limit, i.e.
  fs.aio-max-nr ≥ 131072 for the defaults.
• Opt-in io_uring SQPOLL (--device-uring-sqpoll), one poll thread per ring so submission stays
  parallel across rings.
• Buffer pool depot stripes sized from the machine. The shared overflow depot in
  BufferPool.OriginReturn.cs had a fixed 8 stripes, so on a large box the threads that spill to
  it serialize on those 8 locks. The stripe count now comes from the same ConcurrencySharding
  formula (floor 8, cap 64, rounded to a power of two so the existing mask indexing still
  applies). No other pool constant changes, so the byte budget still bounds retained memory.
• C ABI exception firewall. Every native entry point catches C++ exceptions and maps them to a
  return code; the managed callback never lets an exception escape into the native dispatch loop
  (which would silently kill the drainer thread and hang all subsequent IO). Reporting that failure
  is itself guarded, since a host-supplied logger can throw.
• Lock-free disposal safety. Dispose() publishes disposedFlag and drains in-flight; submit
  and lease paths bump their shard counter then re-check the flag (Dekker), so the native handle
  cannot be freed under an in-flight IO or a non-IO native call. ReadAsync / WriteAsync take a
  second, independently balanced lease around the native call: the IO's own bump is dropped by the
  completion callback on a drainer thread, which can fire before the submitting frame has left
  native code.
• Device creation is exception-safe. If the drain probe or drainer start-up throws, the partially
  initialized handle is destroyed and any started drainers are joined before rethrowing; the handle
  is published only once all drainers are running.
• io_uring drain compatibility. Below kernel 5.11 (IORING_FEAT_EXT_ARG absent) liburing emulates
  a wait timeout by posting a timeout SQE, which mutates the submission queue from the completion
  side. The drainer detects the missing feature at ring init and polls the completion queue instead.

Key Technical Details

Affected types:

• NativeStorageDevice — the managed P/Invoke wrapper; sharded in-flight tracking, slot pool,
  handle leasing, completion draining, smart defaults.
• NativeDeviceOptions / LocalMemoryDeviceOptions (new, DeviceOptions.cs) — group the
  device-type-specific tuning that previously sprawled across positional parameters.
• ConcurrencySharding (new) — the min(2 × ProcessorCount, cap) sizing formula, shared by the
  device in-flight shards and the buffer pool depot stripes.
• BufferPool.OriginReturn.cs — depot stripe count derived from that formula (stripe count only).
• file_linux.{h,cc}, native_device.h, native_device_wrapper.cc, thread.h — native submit /
  completion / thread-registration paths.

New server options (all default to today's behavior): --device-io-contexts,
--device-queue-depth, --device-uring-sqpoll, --device-uring-sqpoll-idle-ms,
--device-aio-max-devices. The pre-existing --device-throttle-limit gains a Native-device
default of 4096 (sized for deep NVMe / io_uring queues) rather than the managed devices' 120.

Docs: new Device Tuning guide
derives every knob and internal constant, including which combinations are actually reachable.

Breaking Change

Devices.CreateLogDevice, LocalStorageNamedDeviceFactory and
LocalStorageNamedDeviceFactoryCreator drop their loose ioBackend / localMemorySegmentSize /
localMemoryRingCapacity parameters in favour of the NativeDeviceOptions /
LocalMemoryDeviceOptions objects. This is a deliberate signature break agreed for this PR
(these entry points are not part of the supported public surface); callers pass an options object
instead.

What NOT to Do (for future agents)

• ❌ Don't tie io_uring ring count to the completion-thread count. Rings must scale with
  submitter concurrency; too few rings serialize submitters on a per-ring lock and cost up to ~3×.
• ❌ Don't add a global in-flight counter or a shared free-slot queue on the submit path. That
  is precisely the cache-line ping-pong this PR removed.
• ❌ Don't let a managed exception escape the device completion callback. It crosses the C ABI
  boundary and silently terminates the drainer thread, leaving every later IO pending forever.
• ❌ Don't raise --device-throttle-limit above 4096 expecting more in-flight IO. Device-wide
  in-flight is bounded by NumShards × MaxPerThreadInFlight; raising the ceiling means growing
  SlotsPerShard, not the throttle.
• ❌ Don't rewrite an SQE after io_uring_submit on the SQPOLL path. The poll thread may have
  already consumed it; the non-SQPOLL no-op-rewrite unwind trick is not valid there.
• ❌ Don't call io_uring_wait_cqe_timeout from the drainer without checking IORING_FEAT_EXT_ARG.
  On kernels that lack it, liburing emulates the timeout by taking an SQE and flushing the submission
  queue — from the completion side, without the submit lock, and with a non-null reserved user_data
  the batch dispatcher would treat as a caller context.
• ❌ Don't compare device/KV benchmark numbers across runs on a busy box. Leftover build daemons
  on the pinned NUMA node depress results by ~30% and have already produced one phantom regression.
  Measure both arms back-to-back.

Testing

• DeviceTests.cs (+117), most cases run against both libaio and io_uring: multi-drainer
  parallel mixed IO, multiple devices across threads with no cross-shard slot reuse, high-throttle
  high-in-flight, throttle-above-max clamping, high-concurrency many-threads no-hang, explicit
  io-context/ring-count with multi-ring fan-out submitted from concurrent threads, explicit shallow
  queue depth exercising ring-full backpressure, SQPOLL round-trip, and Reset / RemoveSegment /
  GetFileSize / TryComplete behavior.
• GarnetServerConfigTests.cs (+101): default propagation and override parsing for every new option.
• Benchmarks: Device.benchmark, KV.benchmark and Resp.benchmark READMEs carry the reproduction
  commands and current measured numbers; benchmark/Resp.benchmark/scripts/nvme-raid0-matrix.sh
  regenerates the RESP matrix.

[Copilot]

Pull request overview

Optimizes Tsavorite’s native Linux I/O path for high-concurrency NVMe workloads through configurable ring topology, sharded bookkeeping, and reduced contention.

Changes:

• Adds independent I/O-context, queue-depth, throttle, and SQPOLL tuning.
• Shards buffer/completion state and introduces affine completion draining.
• Adds configuration plumbing, benchmarks, tests, and tuning documentation.

Reviewed changes

Copilot reviewed 43 out of 55 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
website/sidebars.js	Adds device-tuning guide.
website/docs/getting-started/configuration.md	Documents new settings.
website/docs/dev/device-tuning.md	Adds detailed tuning guide.
test/standalone/Garnet.test/GarnetServerConfigTests.cs	Tests option plumbing.
test/cluster/Garnet.test.cluster/ClusterTestUtils.cs	Adds endpoint failover wait.
test/cluster/Garnet.test.cluster/ClusterManagementTests.cs	Avoids failover test race.
libs/storage/Tsavorite/cs/test/test.hlog/DeviceTests.cs	Updates device-test commentary.
libs/storage/Tsavorite/cs/src/core/Utilities/BufferPool.cs	Stripes buffer free lists.
libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteThread.cs	Uses affine completion draining.
libs/storage/Tsavorite/cs/src/core/Index/CheckpointManagement/LocalStorageNamedDeviceFactoryCreator.cs	Plumbs device options.
libs/storage/Tsavorite/cs/src/core/Index/CheckpointManagement/LocalStorageNamedDeviceFactory.cs	Applies device options.
libs/storage/Tsavorite/cs/src/core/Device/StorageDeviceBase.cs	Extends completion API.
libs/storage/Tsavorite/cs/src/core/Device/NativeStorageDevice.cs	Implements core I/O optimizations.
libs/storage/Tsavorite/cs/src/core/Device/LocalStorageDevice.cs	Adapts completion API.
libs/storage/Tsavorite/cs/src/core/Device/IDevice.cs	Adds affine-drain parameter.
libs/storage/Tsavorite/cs/src/core/Device/Devices.cs	Exposes native tuning options.
libs/storage/Tsavorite/cs/src/core/Allocator/AllocatorBase.cs	Selects affine/all-ring draining.
libs/storage/Tsavorite/cs/benchmark/KV.benchmark/README.md	Updates NVMe guidance.
libs/storage/Tsavorite/cs/benchmark/KV.benchmark/Options.cs	Adds benchmark options.
libs/storage/Tsavorite/cs/benchmark/KV.benchmark/KvOutput.Json.cs	Reports renamed throttle option.
libs/storage/Tsavorite/cs/benchmark/KV.benchmark/KvOutput.Csv.cs	Reports renamed throttle option.
libs/storage/Tsavorite/cs/benchmark/KV.benchmark/KvBenchmark.Setup.cs	Applies tuning options.
libs/storage/Tsavorite/cs/benchmark/Device.benchmark/README.md	Documents RAID-0 results.
libs/storage/Tsavorite/cs/benchmark/Device.benchmark/Program.cs	Configures new device knobs.
libs/storage/Tsavorite/cs/benchmark/Device.benchmark/Options.cs	Adds benchmark flags.
libs/storage/Tsavorite/cs/benchmark/Device.benchmark/BenchWorker.cs	Reduces completion polling.
libs/storage/Tsavorite/cc/src/device/thread.h	Pads thread-ID flags.
libs/storage/Tsavorite/cc/src/device/thread_manual.cc	Defines padded flags.
libs/storage/Tsavorite/cc/src/device/native_device.h	Extends native device interface.
libs/storage/Tsavorite/cc/src/device/native_device_wrapper.cc	Extends native C ABI.
libs/storage/Tsavorite/cc/src/device/file_windows.h	Adapts Windows handler.
libs/storage/Tsavorite/cc/src/device/file_linux.h	Adds ring and SQPOLL support.
libs/storage/Tsavorite/cc/src/device/file_linux.cc	Implements batched affine draining.
libs/server/Servers/GarnetServerOptions.cs	Plumbs server settings.
libs/host/defaults.conf	Defines new defaults.
libs/host/Configuration/Options.cs	Adds CLI/config options.
benchmark/Resp.benchmark/scripts/nvme-raid0-matrix.sh	Adds RAID-0 benchmark matrix.
benchmark/Resp.benchmark/README.md	Documents RESP results.
benchmark/Resp.benchmark/Program.cs	Applies load-thread setting.
benchmark/Resp.benchmark/Options.cs	Adds load-thread option.
benchmark/Resp.benchmark/OfflineBench/RespPerfBench.cs	Removes request allocations.
benchmark/Resp.benchmark/OfflineBench/ReqGen.cs	Caches complete argument arrays.
.github/workflows/native-build.yml	Verifies native exports.

Suppressed comments (2)

benchmark/Resp.benchmark/scripts/nvme-raid0-matrix.sh:111

• The GET client also ignores the configurable PORT, so benchmark passes target port 6379 rather than the server started above when PORT is overridden. Forward --port "$PORT" here too.

      local out; out="$($cli_pin dotnet "$RB" -s --op GET --dbsize "$DBSIZE" \
        --keylength "$KEYLEN" --valuelength "$VALLEN" --client LightClient \
        -t "$t" -b "$REQB" --runtime "$RUNTIME" 2>/dev/null)"

website/docs/dev/device-tuning.md:272

• The constants table does not match the implementation: NumShards is min(2 × ProcessorCount, 32), not a 128–1024 clamp, and ReconcileIntervalMs does not exist. These values materially change the documented memory footprint and contention model.

[Ted Hart (TedHartMS)]

Approved with nit comments

[Copilot]

Pull request overview

Copilot reviewed 43 out of 55 changed files in this pull request and generated 1 comment.

Suppressed comments (8)

libs/storage/Tsavorite/cs/src/core/Device/NativeStorageDevice.cs:959

• PerThreadLimit makes every configured throttle above 4096 ineffective: activeShards is at most 32 and each shard is capped at 128, so both --device-throttle-limit 4096 and 65536 produce the same per-shard limits for every possible active-shard count. This contradicts the new 65536 tuning guidance and prevents the advertised higher aggregate in-flight depth. The shard/slot sizing or cap needs to scale with the effective throttle.
  libs/storage/Tsavorite/cs/src/core/Device/Devices.cs:36
• This public method removes the existing ioBackend, localMemorySegmentSize, and localMemoryRingCapacity parameters and changes the type/order immediately after readOnly. Existing named calls no longer compile, positional calls can bind incorrectly, and already-compiled consumers cannot resolve the old CLR signature. Preserve the original overload and introduce the options-based API through a non-ambiguous overload or new method.

        public static IDevice CreateLogDevice(string logPath = null, DeviceType deviceType = DeviceType.Default, bool preallocateFile = false, bool deleteOnClose = false, long capacity = CAPACITY_UNSPECIFIED, bool recoverDevice = false, bool useIoCompletionPort = false, bool disableFileBuffering = true, bool readOnly = false, int numCompletionThreads = 1, ILogger logger = null, NativeDeviceOptions nativeDeviceOptions = null, LocalMemoryDeviceOptions localMemoryDeviceOptions = null)

libs/storage/Tsavorite/cs/src/core/Index/CheckpointManagement/LocalStorageNamedDeviceFactory.cs:42

• Removing the public constructor's ioBackend parameter changes the established signature; old positional calls now pass an IoBackend where int numCompletionThreads is expected, and named ioBackend: calls fail. This also reintroduces the compatibility problem previously addressed for this constructor. Keep the old constructor as a forwarding overload while exposing NativeDeviceOptions separately.
  libs/storage/Tsavorite/cs/src/core/Index/CheckpointManagement/LocalStorageNamedDeviceFactoryCreator.cs:35
• This public constructor likewise removes ioBackend, breaking existing positional and named callers even though the new options object is appended. Retain the previous signature as a compatibility overload that constructs NativeDeviceOptions, rather than replacing it outright.
  website/docs/dev/device-tuning.md:354
• The configuration page is in docs/getting-started, not docs/dev, so this relative link resolves to a nonexistent dev/configuration page. Point it to the existing sibling-directory path.
  benchmark/Resp.benchmark/scripts/nvme-raid0-matrix.sh:103
• The load command's exit status is ignored because the script does not use set -e or check this invocation. If loading fails, the script still benchmarks GET misses and can publish a high but meaningless “NVMe” throughput table. Abort the configuration when the load command fails (and ideally verify the loaded key count) before entering the measured sweep.

  $cli_pin dotnet "$RB" --port "$PORT" --op MSET --dbsize "$DBSIZE" --keylength "$KEYLEN" --valuelength "$VALLEN" \
    --client LightClient --load-threads 32 -b 4096 --runtime 0 >/tmp/nvme-matrix-load.log 2>&1

libs/storage/Tsavorite/cs/src/core/Utilities/ConcurrencySharding.cs:18

• The stated invariant is impossible on hosts with more than 16 logical processors: the formula caps the count at 32, so it cannot remain at or above 2 × ProcessorCount. Reword this to state that the count scales at two per processor only on small hosts and deliberately allows shard sharing after the cap.
  libs/storage/Tsavorite/cs/test/test.hlog/DeviceTests.cs:1367
• This test does not exercise the claimed multi-ring fan-out: all ReadAsync calls run on the single NUnit thread, while pick_context_index() assigns one affine ring per submitting thread. Consequently all 128 reads target one of the eight rings, so regressions in cross-thread ring assignment or draining the other seven rings remain undetected. Submit from at least eight concurrent threads and verify all completions.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 58 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

libs/storage/Tsavorite/cs/benchmark/Device.benchmark/Program.cs:231

• The benchmark's renamed --device-throttle-limit still does not configure LocalMemory's actual in-flight bound: this options object leaves RingCapacity at 0, which CreateLogDevice converts to 1024, while setting device.ThrottleLimit later has no effect because LocalMemoryDevice does not implement Throttle(). Thus the documented --device-throttle-limit 8192 LocalMemory run still uses a 1024-entry ring. Map the requested limit to a power-of-two ring capacity here, as KV.benchmark does.

                    localMemoryDeviceOptions: new LocalMemoryDeviceOptions { SegmentSize = segSize });

benchmark/Resp.benchmark/Program.cs:336

• --load-threads accepts zero or negative values, but LoadData immediately evaluates DbSize % loadDbThreads and divides by it. A user-provided zero therefore crashes the benchmark with DivideByZeroException instead of a configuration error. Validate this new option before calling LoadData.

                    bench.LoadData(loadDbThreads: opts.LoadThreads, keyLen: keyLen, valueLen: valueLen, numericValue: opts.Op == OpType.INCR);