feat: prevent compose output string overflow

[neilime]

Long-running or verbose Compose commands could accumulate stdout/stderr beyond JavaScript's string limit and fail with RangeError: Invalid string length. This adds maxOutputLength to bound captured output independently for each stream, addressing #257.

• The limit counts UTF-16 code units, defaults to Node.js's buffer.constants.MAX_STRING_LENGTH, and accepts 0 to disable buffering. Callbacks and logging still receive every chunk.
• Stdout retains its beginning; stderr retains its most recent output. Capture uses chunk queues, evicts old stderr chunks, and joins once when the result settles, avoiding repeated copies of the full retained window.
• Required truncated.out and truncated.err flags are present on successful, rejected, and typed results. Commands that parse stdout reject if it was truncated.

The README and API docs cover the option and result flags. The commit uses feat: for the new public API.

await compose.upAll({
  cwd: projectDir,
  callback: (chunk, source) => {
    process.stdout.write(`[${source}] ${chunk}`)
  },
  maxOutputLength: 1_000_000
})

Validation: 42 selected buffering, executable-resolution, and parsing tests pass, including nine new cases for chunk eviction, sustained small-chunk output, and output received during the existing exit delay. yarn build and yarn lint pass (lint reports seven existing warnings). Docker integration tests were not run because the sandbox cannot access the Docker socket.

A local Node.js 26 benchmark using 20,000 stderr chunks of 100 bytes with a 1,000,000-code-unit limit took about 4.5 ms with chunk queues, compared with 1.0–1.3 seconds for the previous implementation (two trials, excluding the fixed exit delay).

[AlexZeitler]

Thanks for picking this up, and for shipping tests and docs with it.

Reviewed at 16e4258.

The approach is right: cap before appending, and leave the callback untouched so streaming consumers still see every chunk. That second part is what actually makes #257 solvable, and I'm glad it has a test.

Four small things I'd like to see before merging:

1. Rename the option. maxBuffer is taken by child_process, where it counts bytes and kills the child with ERR_CHILD_PROCESS_STDIO_MAXBUFFER. Here it counts UTF-16 code units and truncates silently. Both differences will trip someone up. The option is unreleased, so renaming costs one line today and a deprecation cycle later. Something like maxOutputLength says what it measures.

2. Document the streaming case. 0 disables buffering completely: Math.max(0, ...) keeps the zero, and the length check then short-circuits on every chunk. Combined with callback, that is exactly what the reporter in #257 needs. The code supports it, but neither README nor docs/api.md mentions it.

3. Drop the ?. in src/index.ts:100. bufferConstants comes from a static import and is always defined, so the 64 * 1024 * 1024 fallback is unreachable. If it ever did become reachable, it would silently cap far below the real limit.

4. Two more tests. One for 0 buffering nothing, which is the #257 path and currently uncovered. One for the default truncating nothing, which guards the default against a later refactor.

Two more things I'd like your opinion on. Happy to take them as follow-ups if you'd rather keep this PR tight:

5. Nothing on the result says output was dropped. config feeds result.out into yaml.parse (src/index.ts:614), images and stats into JSON.parse (src/index.ts:225 and src/index.ts:782). With a low limit, the caller sees a YAML or JSON syntax error that points at their compose file instead of at their own option. A truncated flag on IDockerComposeResult would be additive and let those call sites throw something meaningful.

6. Truncation keeps the head. For result.err I think the tail is what matters. execCompose rejects with the result (src/index.ts:388), and callers read err to find out why the command failed. Compose writes its progress output to stderr, so on a long run that noise fills the buffer from the front and pushes out the actual error message. Would you consider a trailing window for err while out keeps the head? It is asymmetric, but it matches how the two streams get used.

One last thing, entirely optional and not for this PR: chunk.toString() decodes each chunk on its own, so a multi-byte character split across a stream boundary becomes replacement characters. That predates this change and applies with no limit set at all, so it deserves its own PR. StringDecoder fixes it, and the surrogate-pair edge at the cut point along with it.

[neilime]

@AlexZeitler I've handled your feedbacks

[AlexZeitler]

Thanks, that was quick. Reviewed at ad15291.

The rename, the truncated flags, the assertCompleteOutput guard before every parse and the trailing window for err all look right to me, and the test coverage goes well beyond what I asked for.

Two things came out of a closer look. The first one is on me: the trailing window was my suggestion, so the cost that comes with it is mine to report.

1. The trailing window copies the whole buffer on every chunk once the limit is reached.

The err branch runs for every chunk, not only when something gets dropped:

result.err = output.slice(Math.max(0, output.length + tail.length - maxLength)) + tail

As soon as that offset is nonzero, V8 flattens the buffer and copies it, so each chunk costs the full window width. Plain += stays cheap because V8 builds a rope instead and never walks the existing content.

A standalone script, no dependencies:

const CHUNK = 'x'.repeat(100)
const CHUNKS = 20000

const time = (label, fn) => {
  const t0 = process.hrtime.bigint()
  fn()
  console.log(`${label}: ${Number(process.hrtime.bigint() - t0) / 1e6} ms`)
}

time('plain +=', () => {
  let err = ''
  for (let i = 0; i < CHUNKS; i++) err += CHUNK
})

time('trailing window, limit 1e6', () => {
  const maxLength = 1_000_000
  let err = ''
  for (let i = 0; i < CHUNKS; i++) {
    const tail = CHUNK.slice(-maxLength)
    err = err.slice(Math.max(0, err.length + tail.length - maxLength)) + tail
  }
})

time('chunk ring, limit 1e6', () => {
  const maxLength = 1_000_000
  const parts = []
  let total = 0
  for (let i = 0; i < CHUNKS; i++) {
    parts.push(CHUNK)
    total += CHUNK.length
    while (total - parts[0].length >= maxLength) total -= parts.shift().length
  }
  parts.join('')
})

On Node 24 I get well under a millisecond for +=, around four seconds for the trailing window, and a few milliseconds for the ring. Two megabytes of output in small chunks is not much for the kind of overnight run described in #257.

The default limit is not affected: the offset stays zero, and slice(0) hands back the same string. The cost only shows up once the buffer is full, which is exactly the situation the option exists for.

Keeping the chunks in an array, dropping from the front, and joining once in the exit handler would avoid it. out can use the same structure and simply stop pushing once it is full.

2. For release purposes this is a feat, not a fix.

truncated is required on IDockerComposeResult, and TypedDockerComposeResult now inherits it. Code that only reads results is unaffected. Code that constructs one of these objects, in a mock for instance, gets a compile error. With standard-version the current fix: subject would go out as a patch release.

feat: fits the new public option anyway. I would rather keep the field required and change the subject than make the field optional.

[neilime]

@AlexZeitler I've handled your last feedback. Thanks

[AlexZeitler]

Looks good at 0e015dd.

The chunk ring does the job, and the eviction is more careful than what I sketched: a partially evicted chunk gets trimmed in place instead of dropped, and compacting only once half the array is dead keeps that amortized.

I ran the same measurement as before, with the new appendBufferedOutput transcribed verbatim into the standalone script from my last comment. With a limit of one million code units, the twenty thousand chunks that took around four seconds now take a few milliseconds, and half a million chunks stay well under a tenth of a second. The array stays at the size of the live chunks, so the compaction behaves as intended.

I also walked the eviction loop looking for a way to spin forever or to run head past the end of the array. Neither can happen: empty entries always sit before head, and length is exactly the sum of the retained chunk lengths, so the loop terminates and the join can never exceed the limit.

The new cases cover what I would have asked for anyway: partial eviction, a chunk larger than the limit, surrogate pairs, many small chunks, and output arriving after exit.

Thanks for working through all of this. Happy to merge once CI is green.

[AlexZeitler]

📦 1.5.0 is available on npm now.