Skip to content

cli: ignore trailing data after the last frame, and report corrupted archives - #4840

Closed
Karaammm wants to merge 3 commits into
meta:devfrom
Karaammm:dev
Closed

Karaammm wants to merge 3 commits into
meta:devfrom
Karaammm:dev

Conversation

@Karaammm

Copy link
Copy Markdown

What and why

Both parts follow the discussion on the issue. They are separate commits, so they can be taken
independently.

  1. Trailing data after the last frame (first commit). When valid frames were followed by data that
    is not a frame, zstd stopped with "unsupported format" (4 bytes or more) or "unknown header"
    (1 to 3 bytes) and exit status 1, although every frame had already been decoded. It now
    decompresses all frames, ignores the trailing data and warns:
    zstd: WARNING: <file>: trailing data after the last frame is ignored. The exit status is 0.
    gzip and bzip2 also tolerate trailing data with a warning (gzip exits with 2). Input that never
    contained a frame keeps its errors, and --pass-through still copies the trailing data.

  2. Damaged archives (second commit). To tell that case apart from a genuinely damaged archive, a
    zstd frame that is cut short or damaged now ends with
    zstd: <file>: corrupted or incomplete archive; decoded <N> bytes, after the existing error.
    N counts the bytes decoded and written before the failure, across all frames of the file. It is
    printed for a frame that is cut short and for the errors that mean damaged data (corruption
    detected, checksum mismatch, wrong literals header, wrong source size). Other decoding errors,
    such as dictionary mismatch or a window size over the memory limit, are unchanged.

Notes for reviewers

  • The warning is printed at display level 1, not 2: writing to stdout lowers the level to 1, and
    cat file | zstd -dc (the case in the issue) must still show it. -qq silences it.
  • Exit status 0 for trailing data is a behavior change, with two consequences: zstd -d --rm
    removes the source file in that case (as gzip does), and zstd -t passes.
  • A later frame whose magic number is damaged cannot be told apart from trailing data. Such an
    archive now decompresses only the frames before it, with the warning as the only signal. Before,
    it failed.
  • If you would rather keep a failing status for trailing data, the first commit is the one to
    change. The second commit does not depend on it.
  • gz, xz, lzma and lz4 frames do not get the corruption line, because their decoders do not report
    an exact decoded size.

Tests

tests/cli-tests/decompression/trailing-garbage.sh and corrupted-archive.sh. Both fail without the
change.
dictionaries/dictionary-mismatch.sh shows that unrelated decoding errors are unchanged. The
CLI test suite and tests/playTests.sh pass;

Checklist

  • I searched open pull requests and issues: this problem is not already being addressed.
  • Bug fix: it comes with a reproducer or a test that fails without the fix.
  • Performance change: numbers from repeated runs are included, with CPU, compiler, flags and data.
    A gain in one configuration is not a general gain: see the Performance section of CONTRIBUTING.md.
  • I stated below whether AI or other automated tools were used.

Tools used
Claude Code (Claude Sonnet 5.5) was used to find the code path, write the documentation, and the
tests. I ran the tests locally

@meta-cla meta-cla Bot added the CLA Signed label Oct 12, 2026
@Karaammm Karaammm closed this Oct 13, 2026
@tansy

tansy commented Oct 14, 2026

Copy link
Copy Markdown

Wasn't that good enough?

@Karaammm

Copy link
Copy Markdown
Author

I thought I had to make additional changes. I'll make a new PR right now

@Karaammm

Copy link
Copy Markdown
Author

Uploaded:)

@tansy

tansy commented Oct 14, 2026

Copy link
Copy Markdown

I thought I had to make additional changes. I'll make a new PR right now

Well, yes. I meant that in case of actually corrupted archives must return en error, an non-zero value on exit, and that archive cannot be deleted, as it was not successfully decompressed. It also gives the user chance to have a look and possibly recover that data, which will be impossible if archive is deleted.

Other than that, I have no objections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants