Write outlier messages through the ByteBuf API instead of staging in a heap array

[merlimat]

Problem

Since #12, writeTo() to a non-array buffer (direct, composite) stages the whole message in a heap byte[] scratch and bulk-copies it. The scratch is retained on the instance only up to SCRATCH_RETAIN_MAX (1 MiB); a larger message allocates a fresh exact-size heap array on every serialization — multi-MB G1-humongous allocations, invisible to any accounting sized to the target buffer.

Pulsar always serializes into pooled direct buffers, so any message over ~1 MiB (e.g. a ~4.6 MB CommandGetTopicsOfNamespaceResponse for 8192 partitions) now pays its full serialized size in transient heap per write. This is one of the two 0.8.0 regressions behind the ProxyPatternConsumerBackPressureMultipleConsumersTest OOM seen on apache/pulsar#26256 — the other (retention in clear()) is fixed by #19.

Fix

writeTo() now dispatches three ways:

• heap buffers: written in place through the backing array (unchanged);
• messages ≤ SCRATCH_RETAIN_MAX: retained-scratch composition (unchanged — this is where the Compose serialization into byte[] to avoid per-call Unsafe overhead #12 wins live; the scratch is now always retainable, so the conditional retention is gone);
• larger messages: a generated _writeTo(ByteBuf) writes field by field through the ByteBuf API — allocation-free, without reintroducing the pre-Compose serialization into byte[] to avoid per-call Unsafe overhead #12 sun.misc.Unsafe usage, and (unlike pre-Compose serialization into byte[] to avoid per-call Unsafe overhead #12) working for CompositeByteBuf targets. Nested messages write through as well, so no element of the tree stages in a scratch array; bulk data still moves via bulk getBytes()/writeBytes(), so only tags and varints pay the per-call ByteBuf overhead — irrelevant at multi-MB sizes.

The ByteBuf write helpers this reuses (writeVarInt, writeString, …) were kept by #12 and encode byte-identically to the array writers: same ASCII test, and computeStringUTF8Size already uses Netty's utf8Bytes, so the UTF-8 fallback agrees with reserveAndWriteUtf8 byte for byte.

Verification

• LargeWriteThroughTest: byte-identity against protobuf-java for >1 MiB messages on direct and composite targets (repeated strings incl. non-ASCII, bytes payloads, nested-message trees, and the exact Pulsar BaseCommand shape, which also exercises the bit-driven traversal variant); scratch retention just below the threshold; and — via ThreadMXBean.getThreadAllocatedBytes — that 5 writes of a multi-MB message allocate less than a quarter of one message size.
• JMH (interleaved A/B vs 0.8.0, JDK 21, Apple M-series): small-message serialize unchanged.
• End-to-end together with Release data references in clear() for messages above CLEAR_RETAIN_MAX #19 (0.8.1-SNAPSHOT in Pulsar's gradle build): the proxy back-pressure test goes from OOM at 275/500 requests (free-heap floor 1–7 MB) to 500/500 in ~8 s (min free heap 97 MB).

Independent of #19; the two merge cleanly in either order (verified by merge simulation plus the full suite on each branch).

[merlimat]

Superseded by #20: instead of falling back to per-field ByteBuf-API writes above the scratch cap, direct buffers above 512 bytes are written in place through their NIO view (no scratch, no copy, at any size), which measured 21–44% faster than the scratch path from 600 B to 100 KB and matches this PR's numbers on the multi-MB cases, while the tiny hot-path messages keep the unchanged scratch path. Closing in favor of #20.