Due to how GH is built, when a job dependency fails, said job is skipped by default. Then, a skipped Checks API report shows up gray in the checks widgets of pull requests.
The branch protection rules on GH have an unobvious quirk — skipped jobs count the same as successful ones. This means that when the test matrix fails, the branch protection will let pull requests get merged. Just look at this PR, for example: #73.
To solve this, you have to run the all-checks always and make the dependency outcome checks in it.
I have a GitHub Action that solves this problem: https://github.com/marketplace/actions/alls-green#why. It unifies the solution and is easier to maintain. I've got it integrated in a bunch of places in the Python ecosystem: in aio-libs, PyCA, PyPA, CPython, pytest etc. (off the top of my head)
You can find usage examples in those repos or any of those.
Due to how GH is built, when a job dependency fails, said job is skipped by default. Then, a skipped Checks API report shows up gray in the checks widgets of pull requests.
The branch protection rules on GH have an unobvious quirk — skipped jobs count the same as successful ones. This means that when the test matrix fails, the branch protection will let pull requests get merged. Just look at this PR, for example: #73.
To solve this, you have to run the
all-checksalways and make the dependency outcome checks in it.I have a GitHub Action that solves this problem: https://github.com/marketplace/actions/alls-green#why. It unifies the solution and is easier to maintain. I've got it integrated in a bunch of places in the Python ecosystem: in aio-libs, PyCA, PyPA, CPython, pytest etc. (off the top of my head)
You can find usage examples in those repos or any of those.