Skip to content

1.100 beta crater regression: "No space left on device" and massive slowdown during doctests #164145

New issue
New issue
Closed
#164202
Closed
1.100 beta crater regression: "No space left on device" and massive slowdown during doctests#164145
#164202
Labels
A-doctestsArea: Documentation tests, run by rustdocC-bugCategory: This is a bug.S-has-bisectionStatus: A bisection has been found for this issueS-has-mcveStatus: A Minimal Complete and Verifiable Example has been found for this issueT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.T-rustdocRelevant to the rustdoc team, which will review and decide on the PR/issue.regression-from-stable-to-betaPerformance or correctness regression from stable to beta.

Description

@theemathas
theemathas
opened on Oct 11, 2026

View all comments

This regression was found in the 1.100 beta crater run.

The fixed_star_frame-1.29.1 consistently has its tests pass in crater on stable rust. However, it consistently fails in crater on beta rust.

It appears to me that we're doing something much less efficiently on beta than on stable, and using much more disk space.

I tried running the tests on the dev desktop (dev-desktop-eu-2.infra.rust-lang.org). cargo +stable test used 0m32.266s to complete. However, cargo +beta test used 1m11.525s to complete. That is, running tests on beta is over twice as slow.

Activity

  1. theemathas
    added
    regression-from-stable-to-betaPerformance or correctness regression from stable to beta.
    C-bugCategory: This is a bug.
    E-needs-bisectionCall for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc
    E-needs-mcveCall for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example
    on Oct 11, 2026
  2. rustbot
    added
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Oct 11, 2026
  3. theemathas
    added a parent issue on Oct 11, 2026

  4. theemathas commented on Oct 11, 2026

    @theemathas
    theemathas
    on Oct 11, 2026
    ContributorAuthor

    Some ongoing discussion at #general > How to triage a massive test-running perf regression?

  5. theemathas commented on Oct 11, 2026

    @theemathas
    theemathas
    on Oct 11, 2026
    ContributorAuthor

    Bisected with cargo bisect-rustc --start=1.99.0 --end=beta --timeout 50 -- test:

    ********************************************************************************
Regression in 5392d2f545c6836dc79f209bcc14ac7179dbc4f2
********************************************************************************

Attempting to search unrolled perf builds
Found commits ["0e21efee", "0aebea15", "770a9cd8", "449ef23a", "a95bb512", "5fd80363", "a49eccce", "5ba40ad4", "2983377a", "033e87fb", "3b668784", "16c23a49"]
installing 0e21efee1ad09fb750d7925735bc0495cc10e971
rust-std-nightly-x86_64-unknown-linux-gnu: 31.40 MB / 31.40 MB [=============================================================================] 100.00 % 11.56 MB/s testing...
RESULT: 0e21efee1ad09fb750d7925735bc0495cc10e971, ===> Successfully compiled
uninstalling 0e21efee1ad09fb750d7925735bc0495cc10e971

installing 0aebea155e1762d121a36050809579171759e979
rust-std-nightly-x86_64-unknown-linux-gnu: 31.38 MB / 31.38 MB [=============================================================================] 100.00 % 11.47 MB/s testing...
RESULT: 0aebea155e1762d121a36050809579171759e979, ===> Compile error
uninstalling 0aebea155e1762d121a36050809579171759e979

Regression in https://github.com/rust-lang/rust/commit/0aebea155e1762d121a36050809579171759e979. Note that if it is a legacy rollup build, it might be available in https://github.com/rust-lang-ci/rust/commit/0aebea155e1762d121a36050809579171759e979.
The PR introducing the regression in this rollup is #161868: libtest: never iterate over all tests in `--exact` mode

    Regressed in #161868. cc @RalfJung (author), @Mark-Simulacrum (reviewer)

  6. theemathas
    added
    A-libtestArea: `#[test]` / the `test` library
    T-libsRelevant to the library team, which will review and decide on the PR/issue.
    S-has-bisectionStatus: A bisection has been found for this issue
    and removed
    E-needs-bisectionCall for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc
    on Oct 11, 2026
  7. github-project-automation
    added this to testing-devex backlogon Oct 11, 2026
  8. theemathas
    added
    A-doctestsArea: Documentation tests, run by rustdoc
    on Oct 11, 2026

  9. 24 remaining items

  10. RalfJung commented on Oct 12, 2026

    @RalfJung
    RalfJung
    on Oct 12, 2026 via email
    Member

  11. GuillaumeGomez commented on Oct 12, 2026

    @GuillaumeGomez
    GuillaumeGomez
    on Oct 12, 2026
    Member

    Found an idea: the temp dirs are removed once all tests are run, so I'll have two doctests in 2021 edition run with --test-threads=1 and if the count is more than 1 when the second doctest runs, then the bug is there (already wrote the regression test). So now let me check how to fix the behaviour change.

  12. GuillaumeGomez commented on Oct 12, 2026

    @GuillaumeGomez
    GuillaumeGomez
    on Oct 12, 2026
    Member

    You didn't add a regression test we could use for ensuring that my fix is not regressing cargo miri nextest?

  13. RalfJung commented on Oct 12, 2026

    @RalfJung
    RalfJung
    on Oct 12, 2026
    Member

    I don't know how to add a regression test checking time complexity... the test is "make sure that with 1000s of #[test] functions it still can run a single test in Miri in reasonable time".

  14. GuillaumeGomez commented on Oct 12, 2026

    @GuillaumeGomez
    GuillaumeGomez
    on Oct 12, 2026
    Member

    Created #164202.

  15. teor2345 commented on Oct 12, 2026

    @teor2345
    teor2345
    on Oct 12, 2026
    Member

    If it's quadratic then maybe 10K or 100K macro-generated tests would run in a few seconds with the fix, or timeout with the bug?
    Not a very specific test (which might be good if we're testing for anything quadratic), but we'd notice if it added an hour or more to the test time.

  16. RalfJung commented on Oct 12, 2026

    @RalfJung
    RalfJung
    on Oct 12, 2026 via email
    Member
  17. JonathanBrouwer
    added 3 commits that reference this issue on Oct 12, 2026

    Rollup merge of rust-lang#164202 - GuillaumeGomez:fix-doctest-temp-re…

    58aaa4c

    Rollup merge of rust-lang#164202 - GuillaumeGomez:fix-doctest-temp-re…

    ab02446

    Rollup merge of rust-lang#164202 - GuillaumeGomez:fix-doctest-temp-re…

    0bfc997
  18. rust-bors
    closed this as completedin 0bfc997on Oct 12, 2026
  19. github-project-automation
    moved this to Done in testing-devex backlogon Oct 12, 2026
  20. rust-bors
    added a commit that references this issue on Oct 12, 2026

    Unrolled build for #164202 in rollup 164216

    7e6c6cb
  21. apiraino
    removed
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Oct 13, 2026
  22. BenMac31
    added a commit that references this issue on Nov 2, 2026

    Rollup merge of #164202 - GuillaumeGomez:fix-doctest-temp-removal, r=…

    cf43097
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-doctestsArea: Documentation tests, run by rustdocC-bugCategory: This is a bug.S-has-bisectionStatus: A bisection has been found for this issueS-has-mcveStatus: A Minimal Complete and Verifiable Example has been found for this issueT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.T-rustdocRelevant to the rustdoc team, which will review and decide on the PR/issue.regression-from-stable-to-betaPerformance or correctness regression from stable to beta.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions