Skip to content

Improve github actions ci build and runtime behavior - #409

Merged
linearcombination merged 7 commits into
WycliffeAssociates:doc-dev.walink.orgfrom
linearcombination:limit-frontend-test-time
Nov 17, 2026
Merged

linearcombination merged 7 commits into
WycliffeAssociates:doc-dev.walink.orgfrom
linearcombination:limit-frontend-test-time

Conversation

@linearcombination

@linearcombination linearcombination commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #410

Problem

Frontend e2e tests were failing CI by hitting the 35-minute step limit.
The full Playwright suite ran serially (workers: 1) on a single runner
alongside the whole stack, and each test job also rebuilt every image.
All images shared one GHA cache scope, so they evicted each other's
layers and builds rarely hit the cache.

Changes

Builds

  • Build each image once in a matrix job (api, frontend, usfmparser,
    frontend-tests, fileserver) and pass it to test jobs as an artifact,
    instead of rebuilding per job.
  • Give each image its own GHA cache scope.
  • Remove setup-qemu-action (no multi-platform builds) and
    github-env-vars-action from jobs that don't use CI_* vars.

Tests

  • Shard Playwright across 4 parallel runners via PLAYWRIGHT_SHARD
    (testsDockerfile + docker-compose.frontend-test.yml). Defaults to
    1/1, so local runs still execute the full suite.
  • Add an untimed "Prepare compose stack" step so the 35-minute step only
    runs tests.
  • Dump compose logs on failure.

Other

  • Cancel superseded runs on the same branch.
  • curl --fail on the deploy webhook.

Results

Before After
Frontend tests timed out at 35 min 6 min (slowest shard)
Total workflow sometimes cancelled due to exceeding time limit, highly variable 17 min

Production / DevOps impact

None intended. Push-Images builds and pushes the same images, tags and
build args as before; the deploy webhook is unchanged. Changes are
limited to how images are built and tested in CI.

Notes for reviewers

  • Job timeouts changed: frontend 60 → 45 min per shard; API job now
    45 min (previously unset). Tuned from observed durations plus some extra time to account for variations in backend API performance or github traffic loads/queueing.
  • doc-fileserver:latest relies on the compose project name doc
    (the checkout directory). Renaming the repo requires updating the tag.

Limit the total time that frontend tests can run as otherwise github
actions ci can sometimes run up to their 6 hour max before they cancel
it. There are various reasons why this can happen on github actions ci
when it doesn't ever happen locally for the same tests. We don't want
to wait around so long when it does happen. Better to fail "early",
e.g., within 60 minutes
@linearcombination
linearcombination changed the base branch from doc.bibleineverylanguage.org to doc-dev.walink.org November 16, 2026 20:59
@linearcombination linearcombination self-assigned this Nov 16, 2026
playwright.config.ts

Since gh actions ci runs in Docker, we have to flag that so that it
doesn't use too many resources. Locally we can still run tests, if
desired, with npx playwright tests and in that case
playwright.config.ts would yield CI false and thus would allow a more
generous use of resources and concurrency on a local dev machine.
The frontend test step hit its 45-minute limit: the full Playwright suite
runs serially (workers: 1) on a single runner alongside the whole stack.

Workflow (docker-publish.yml):
- Build each image once in a matrix job, save it as a tarball artifact,
  and docker load it in the test jobs instead of rebuilding it per job.
- Give each image its own GHA cache scope; they previously shared the
  default scope and evicted each other's layers.
- Split the frontend tests across 4 matrix shards, each on its own runner.
- Run compose with --no-build --pull never so the timed step only runs
  tests.
- Dump compose logs on test failure.
- Cancel superseded runs on the same ref via a concurrency group.
- Drop setup-qemu-action (no multi-platform builds) and remove
  github-env-vars-action from jobs that don't use CI_* vars.
- Simplify the Push-Images condition to github.event_name == 'push'.
- curl --fail on the deploy webhook.

docker-compose.frontend-test.yml:
- Pass PLAYWRIGHT_SHARD through to the test runner (defaults to 1/1).

frontend/testsDockerfile:
- Run `playwright test --shard="$PLAYWRIGHT_SHARD"`, defaulting to 1/1
  so local runs still execute the full suite.
The first sharded run failed before any test started:
`up --no-build --pull never` couldn't find doc-fileserver:latest
(fileserver has only `build: ./web`, no image tag), and would next
have failed on redis and mher/flower, which are pulled from Docker Hub.

- Build fileserver in the image matrix, tagged doc-fileserver:latest
  (the <project>-<service> name compose expects), and load it in both
  test jobs.
- Add an untimed "Prepare compose stack" step (`up --no-start`) that
  pulls redis/flower and builds anything still missing, so the timed
  step only starts containers and runs tests.
- Drop --pull never from the test steps; keep --no-build so a missing
  image fails fast instead of building inside the 35-minute step.
@linearcombination linearcombination changed the title Limit frontend test time Improve github actions ci build and runtime behavior Nov 17, 2026
@linearcombination linearcombination moved this from In progress to In Review in DOC Project Management Nov 17, 2026
@linearcombination
linearcombination merged commit ad0f18a into WycliffeAssociates:doc-dev.walink.org Nov 17, 2026
13 checks passed
@github-project-automation github-project-automation Bot moved this from In Review to Done in DOC Project Management Nov 17, 2026
@linearcombination
linearcombination deleted the limit-frontend-test-time branch November 17, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants