Skip to content

[Feature] Smoke test failures do not fail the PR #2885

Description

@EmmaYuan1015

Problem

The smoke test step in .ci/smoke-tests-common-validation.yml uses
continueOnError: true.

As a result, smoke test failures do not fail the PR pipeline. This can allow a
PR with failing smoke tests to appear successful.

Removing continueOnError alone may prevent diagnostic screenshots from being
published when a smoke test fails because subsequent steps do not run by
default.

Proposed Change

  • Remove continueOnError: true from the smoke test step so failures propagate
    to the pipeline result.
  • Add condition: always() to the screenshot publishing step so diagnostic
    screenshots are still uploaded after a failure.

Expected Behavior

  • A smoke test failure causes the corresponding pipeline job to fail.
  • Smoke test screenshots are published regardless of the test result.

Acceptance Criteria

  • The pipeline fails when npm run smoke-tests exits with a non-zero status.
  • The screenshot publishing step runs after both successful and failed smoke
    test executions.
  • Successful smoke test runs continue to pass without behavioral changes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions