Skip to content

Promote Playwright CI jobs from non-blocking to blocking

Problem

Three Playwright jobs in .github/workflows/ci.yml still run with continue-on-error: true:

  • e2e-lms
  • e2e-lms-auth-critical
  • e2e-cms

This is intentional while stabilization is in progress, but there is no explicit graduation bar for returning these checks to merge-blocking status.

Proposal

Define a measurable promotion gate and a rollout order.

Promotion metrics

Evaluate each job over a rolling 14-day window on main:

  • Pass rate >= 98% (success / total runs).
  • No unresolved deterministic regression older than 72 hours (tracked via linked issue/ticket).
  • Flake rate <= 2% (failures that pass on rerun) using Playwright JSON report artifacts.

Promotion order

  1. e2e-lms-auth-critical (already uses --retries=0 and targeted scope).
  2. e2e-cms.
  3. e2e-lms full suite.

Implementation steps

  1. Track daily pass/fail counts for each job from workflow history and artifacts.
  2. Once a job meets the bar for 14 consecutive days, flip that job's continue-on-error to false.
  3. Keep the other jobs non-blocking until they independently meet the bar.

Acceptance criteria

  • Promotion criteria are documented in this ticket and referenced from T-017.
  • The original 14-day pass-rate/flake-rate gate is recorded as superseded by the repo-wide decision to make lint/check/test/e2e strictly blocking in one change.
  • e2e-lms-auth-critical is promoted by setting continue-on-error: false.
  • e2e-cms is promoted by setting continue-on-error: false.
  • e2e-lms is promoted by setting continue-on-error: false.

Implemented direction

  1. Removed continue-on-error: true from all three Playwright jobs in .github/workflows/ci.yml: e2e-lms, e2e-lms-auth-critical, and e2e-cms.
  2. Updated failure summaries in each job to reflect blocking behavior (Job is blocking; failures fail the workflow.).
  3. Promoted all three jobs in one implementation pass instead of the staged order defined in this proposal because CI policy was hardened globally to fail on any lint/check/test/e2e failure.

Notes / decisions log

  • 2026-08-11: Ticket closed as done. All three Playwright jobs are now blocking in CI with continue-on-error removed.
  • 2026-08-11: The original staged promotion gate (14-day pass/flake window before each promotion) was superseded by the strict CI failure-gate decision and implemented in a single rollout.

Out of scope

  • Rewriting Playwright suites.
  • Changing app runtime/test architecture.
  • Introducing a new test-reporting service.