Repository navigation
macOS builders randomly stops working on GitHub Actions #71988
Description
We started noticing that occasionally one of our macOS builders stop working, marking the job as either failed or canceled and providing no logs for the build. Examples of such failed builds are:
- Rollup of 6 pull requests #71934 - https://github.com/rust-lang-ci/rust/runs/648528503
- Rollup of 7 pull requests #71951 - https://github.com/rust-lang-ci/rust/runs/649952440
- Update cargo #71925 - https://github.com/rust-lang-ci/rust/runs/651724911
- upgrade chalk and use chalk-solve/chalk-ir/chalk-rust-ir #69406 - https://github.com/rust-lang-ci/rust/runs/656100319
- [WIP] Rollup of 5 pull requests #72083:
Activity
emilyalbini commentedon May 8, 2020
New one: https://github.com/rust-lang-ci/rust/runs/656100319
emilyalbini commentedon May 11, 2020
Two new ones in a row from #72083:
RalfJung commentedon May 14, 2020
This broke the 2nd rollup in a row: #72187
Makes it kind of hard to land anything, really.^^
There's no error, just a red cross next to "upload artifacts to S3".
emilyalbini commentedon May 14, 2020
We have a call with Github today, if there is no news we'll rollback the gate on GHA.
emilyalbini commentedon May 14, 2020
GitHub is going to continue looking into this ❤️
In the meantime I'll remove the double-gate, to let the queue run more smoothly.
emilyalbini commentedon May 14, 2020
Removed the gate on GHA.
RalfJung commentedon May 14, 2020
@pietroalbini thanks!
Does this take effect on the already running #72202 (comment) ?
emilyalbini commentedon May 14, 2020
It should.
Aaron1011 commentedon May 18, 2020
Possibly related: https://github.com/rust-lang-ci/rust/runs/685622726
emilyalbini commentedon Jul 3, 2020
Just got another failure related to this:
ci: turn of macos for github actions
youknowone commentedon Jul 24, 2020
I am getting similar experience in other rust project, so happy to see Github is looking into this issue.
thejoebourneidentity commentedon Jul 29, 2020
Thanks to everyone for their patience on this issue, I work on the GitHub Actions compute platform that powers our runners and wanted to share an update.
We're aware of the issue and have multiple members of our team across our entire hardware stack actively working on it. We've been able to identify a specific data center that may be the primary failure point. We're rapidly building out additional telemetry and alerting to help us find the root cause. Stay tuned, we want to fix this as soon as possible.
matthewmccullough commentedon Jul 29, 2020
@thejoebourneidentity Thanks for the insight into the bug investigation and reiteration that you and the team are working on it.
alepauly commentedon Aug 27, 2020
@pietroalbini We continue looking at this and I wanted to confirm you are still seeing the exact same symptoms of disconnects. We've been able to curve the cases where the runner disconnects and are now focusing more on perf issues that might be leading to timeouts. I browsed a few workflows and noticed timeouts such as this. Is this a good illustration of the current problem or are you seeing disconnects at a high rate still? Thanks!
emilyalbini commentedon Aug 27, 2020
@alepauly thanks for getting back to us! Indeed, it seems like the runner disconnects are not appearing anymore! I'll monitor closely the build results in the coming week to see if they appear.
Regarding the cancelled builds, like the one you mentioned, that's a mistake on our end 😅. Since our build matrix is so big we built an action to cancel builds once a new commit is pushed to the branch. While this is not a problem normally since our queue bot only pushes a new commit to the test branch when the previous one finishes, we're not gating our builds on the macOS builders yet due to this issue, so the queue bot isn't waiting for them to finish before pushing the new commit (causing the action to cancel the build).
I opened a PR to disable the build cancellation for the macOS jobs, so the cancellations should disappear as soon as the PR lands. If we don't see any other spurious failure for a while we'll try to gate our builds on macOS again!
Thanks again for looking into this issue ❤️
Auto merge of rust-lang#75995 - pietroalbini:ci-let-fallible-finish, …
emilyalbini commentedon Sep 3, 2020
It seems a runner disappeared only one time across the whole week, which is definitely an improvement!
I'll propose during the next weekly infrastructure team meeting to try and double-gate macOS on both Azure and GitHub Actions, to see what the impact would be on our queue size.
alepauly commentedon Sep 3, 2020
We'll look at it on our end, we expect this to be better but not fully solved yet. Thanks for the report.
Auto merge of rust-lang#76755 - pietroalbini:gha-macos, r=Mark-Simula…
emilyalbini commentedon Nov 16, 2020
We didn't see it anymore for months, closing this. Thanks to everyone involved in fixing this! ❤️