Repository navigation
rustc hangs when generating very large arrays #81188
Description
I tried this code:
const VALUES: [usize; std::u32::MAX as usize] = {
todo!();
}It seems that generating large arrays causes rustc to start using up a LOT of memory - eats up over 32gb for this.
Meta
rustc --version --verbose:
rustc 1.49.0 (e1884a8e3 2020-12-29)
binary: rustc
commit-hash: e1884a8e3c3e813aada8254edfa120e85bf5ffca
commit-date: 2020-12-29
host: x86_64-unknown-linux-gnu
release: 1.49.0
Activity
jonas-schievink commentedon Jan 19, 2021
You are creating an array containing 4 billion elements, the compiler isn't too happy about that
[-]rustc hangs when using lazy_static to generate array of structs containing std::os::unix::io::RawFd[/-][+]rustc hangs when generating large arrays[/+][-]rustc hangs when generating large arrays[/-][+]rustc hangs when generating very large arrays[/+]cdmistman commentedon Jan 19, 2021
Makes sense - still,
clangis able to compile similar programs with no problem and much less memory usage.jyn514 commentedon Jan 19, 2021
I think this is a duplicate of #81124.
jyn514 commentedon Jan 19, 2021
A similar program causes rustc to abort:
Details
jyn514 commentedon Jan 19, 2021
Relevant code:
rust/compiler/rustc_mir/src/const_eval/eval_queries.rs
Line 49 in c5a96fb
jyn514 commentedon Jan 19, 2021
Not sure who to ping, maybe @oli-obk - would it be possible to only allocate places if they're something other than a constant value? Something like this:
oli-obk commentedon Jan 20, 2021
cc @rust-lang/wg-const-eval we could delay the allocation of local
_0(return value) until its first use (and its only use, now that we have mir_for_ctfe). This would allowto actually show the panic and not run out of memory or hang when called
RalfJung commentedon Jan 20, 2021
First of all, even if we end up supporting the case where the function panics, the OOM will still occur once you actually implement the function. So I'd say the benefit of supporting this is rather marginal.
On the cost side, the interpreter is already complicated enough as things are; every extra bit of complexity will increase the risk of bugs. If we do anything here, I think the right layer to do that is
Allocation: we can delay actually allocating theVec<u8>byte buffer until the first write.This program asks the compiler to allocate
size_of::<usize>() * 4GBof RAM... I'd say if people want to shoot themselves into the foot we should let them. I don't see anything reasonable we can do about code like this.oli-obk commentedon Jan 20, 2021
Specifically this example allocates an all-zero static allocation, which is guaranteed immutable. This kind of static allocation has custom support in LLVM and won't take up any memory until runtime. So if we can track this kind of information through the compiler, we can support that, too. But all of this is a big rabbit hole.
Related issues:
A previous attempt for undef bytes: #62655
This issue is still actionable by itself, as just the action of allocating an all-undef array should not take very long. It should either OOM quickly or manage to allocate that array. It should only be marginally slower than
vec![0_usize, u32::max_value() as usize]on the same machine at runtime. So we need to find out where exactly it is "stuck" on a machine with sufficient memory to actually show this.@cdmistman can you run a rustc in
gdband give us a backtrace after a minute of compiling your example?RalfJung commentedon Jan 20, 2021
This is not an all-undef array though, it is an all-zero array. A lot more extra hackery is needed to add "lazy zeros" everywhere throughout CTFE. I assume writing all these zeros is what takes so much time.
oli-obk commentedon Jan 20, 2021
The original issue
is an all undef array, because we create the array before running the initializer body. Otherwise I would agree with the assessment that interpreting 2^32 index + assignment ops (even if not in actual MIR ops, but just a loop in the interpreter) would be the issue here.
tesuji commentedon Jan 20, 2021
Consider this Rust program (file named
hang.rs):Run
rustc hang.rs. Then I have the following backtrace through gdb.Backtrace
oli-obk commentedon Jan 20, 2021
Thanks, but that's the program that will actually initialize the bits with all zeros, if you replace the initializer body
[0; u32::MAX as usize]withtodo!(), does it also "hang"?tesuji commentedon Jan 20, 2021
Rustc doesn't hang in that case. I just have this error from rustc:
Also run through
time -vI got:oli-obk commentedon Jan 20, 2021
Oh, then I don't know what happened to the original example of @cdmistman Maybe your OS started using swap space and that slowed everything down?
I'm going to close this issue as not actionable then.
Thanks @lzutao for investigating
RalfJung commentedon Jan 20, 2021
The original issue was
So, it's about RAM usage, not hanging.
[usize; 2^32]is indeed 32GB in size (on a 64bit system). So the fix here would be to makeAllocationallocate itsbyteslazily.oli-obk commentedon Jan 20, 2021
Oh... I read this as "start using.." -> "never stops"... But yea, that seems totally fine to me, as in the happy path you'd also use that much memory. I don't think we need to do anything here except for the discussed things in the actually successfully compiling path for which we already have issues.