Repository navigation
tests: skip --max in levels.sh on low-memory Linux systems - #4784
Open
JohannesGezachew wants to merge 1 commit into
Open
JohannesGezachew wants to merge 1 commit into
JohannesGezachew wants to merge 1 commit into
Conversation
`zstd --max` needs about 8.5 GB of memory, so compression/levels.sh gets OOM-killed (exit 137) on machines, VMs and containers with less memory than that. On Linux, read the physical memory from /proc/meminfo and lower it to the cgroup v2/v1 memory limit when there is one, since /proc/meminfo reports the host's memory inside a container. Skip the --max step when less than 9 GiB is available, the same way it is already skipped on 32-bit builds. Other platforms are unaffected. Fixes meta#4748
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4748.
compression/levels.shalways runszstd --max, which needs about 8.5 GB of memory. On machines, VMs and CI containers with less than that, the test is OOM-killed and fails:This skips the
--maxstep on Linux when less than 9 GiB is available, the same way it is already skipped on 32-bit builds (file-max.zstis a copy offile-19.zst, so the rest of the test still runs).Available memory is the
MemTotalfrom/proc/meminfo, lowered to the cgroup memory limit when there is one (/sys/fs/cgroup/memory.maxfor cgroup v2,memory/memory.limit_in_bytesfor v1). The cgroup part matters because inside a container/proc/meminfoshows the host's memory: in a container limited to 4 GiB on an 18 GiB VM it still reportsMemTotal: 18388764 kB, so checking/proc/meminfoalone would not help there.The detection runs inside the existing
set +vblock, so it is not echoed. Only the newelifbranch shows up inlevels.sh.stderr.exact, and it is the same on every platform. On other platforms,mem_kibstays empty and--maxruns as before. That keeps it running on the macOS CI runners, where memory compression lets it pass with less RAM.Testing
zstd --maxon the test's 64 KiB input peaks at 8,397,836 KiB RSS on Linux (/usr/bin/time -v). Withdocker run --memory=X --memory-swap=Xit is killed at 8 GiB and succeeds at 9 GiB, hence the 9 GiB threshold.levels.shon Linux (Debian bookworm,/bin/shis dash), aarch64:mem_kib--maxOn macOS (arm64),
mem_kibis empty,--maxstill runs, and all 44 cli-tests pass. I also checked the script withsh -nunder dash,bash --posixand macOS/bin/sh.I could not test the cgroup v1 path, because Docker Desktop only uses cgroup v2. On v1, an unlimited group reports a very large number, which is never lower than
MemTotal, so it is ignored.