Skip to content

E2E test stability & CI re-enablement

Problem

Playwright E2E tests are disabled in CI (the E2E jobs are commented in .github/workflows/ci.yml). Regressions in critical user flows (auth, enrollment, content editing, admin CRUD) can therefore escape until post-merge or production checks.

The original ticket statement is now partially stale:

Current stabilized baseline:

  • waitForTimeout debt in workarea specs has been removed.
  • LMS and CMS Playwright startup now wait on /api/health.
  • CI is re-enabled with blocking E2E jobs and JSON report artifacts.
  • CMS setup-project auth now uses API-level credentials bootstrap (CSRF + callback) and no longer depends on flaky UI-form interactions.

Proposal

Codify and enforce the following E2E stability contract.

1) Wait strategy (mandatory)

  • Use web-first assertions and retry-aware helpers only:
    • expect(locator).toBeVisible()/toHaveCount()/toContainText()
    • expect.poll(...)
    • expect(async () => ...).toPass()
  • Reuse list settling helpers in apps/cms/test/e2e/list-search-helpers.ts for debounced/filtering list surfaces.
  • Ban page.waitForTimeout(...) in specs.
  • Avoid waitUntil: "networkidle" and visibility-based if branching as synchronization primitives.

2) Startup/readiness contract

  • Both Playwright webServer entries must wait on /api/health endpoints from T-023, not only root/base URLs.
  • Keep hermetic startup commands (db reset/migrate/seed) and pinned env variables for deterministic CI runs.

3) Playwright config parity contract

  • CI retries stay at 2; local retries remain 0.
  • forbidOnly enabled in CI.
  • Keep trace: "on-first-retry" and screenshot: "only-on-failure".
  • Keep explicit test and expect timeouts.
  • Maintain auth bootstrap with setup project + storageState where applicable.

4) Reporter + flaky telemetry contract

  • CI reporter output must include:
    • Human-readable output (line and/or html)
    • Machine-readable JSON report (playwright-report/results.json) for flaky tracking.
  • Upload report artifacts in CI for both apps.

5) CI rollout contract

  • Re-enable E2E as separate jobs for LMS and CMS.
  • Start non-blocking (continue-on-error: true) while stabilization is in progress, then promote to blocking (continue-on-error: false) once CI policy allows.
  • Emit clear failure summaries in GitHub job output.
  • Track flaky rate from JSON reports for 2 weeks, then promote to blocking once stable (promotion gate tracked in T-030).
  • Post-promotion flaky-rate monitoring remains owned by this ticket until a dedicated follow-up ticket is created and linked.

Acceptance criteria

  • Ticket moved from tickets/deferred/ to active tickets/ and marked in-progress
  • E2E jobs are enabled in .github/workflows/ci.yml as separate blocking jobs
  • Zero waitForTimeout() calls in CMS/LMS E2E specs
  • LMS and CMS Playwright webServer.url point to /api/health
  • CI produces JSON Playwright report (playwright-report/results.json) and uploads report artifacts
  • npm run check passes
  • CMS setup project auth flow (auth.setup.ts) is stable for all seeded roles in local smoke

Implemented / hardening pass (2026-07-10)

Baseline before this pass:

  • E2E CI jobs were commented out in .github/workflows/ci.yml.
  • CMS already used /api/health readiness in Playwright config.
  • LMS still used base URL readiness.
  • CMS workarea specs still contained debounce sleeps.

This ticket now standardizes the explicit anti-flakiness contract and tracks implementation status against concrete files.

Out of scope

  • Adding new E2E test coverage (separate initiative).
  • Visual regression snapshots/screenshots as a testing strategy.
  • E2E sharding/parallelization optimization (defer until stability target is met).

Notes / decisions log

  • 2026-07-07: Proposed. Health check endpoint (T-023) available as startup signal.
  • 2026-07-09: Deferred — moved to tickets/deferred/.
  • 2026-07-10: Reactivated and moved back to active tickets as in-progress. Scope updated to reflect current repo state (existing webServer configs and implemented health endpoints), and expanded with codified wait/reporting/CI rollout contracts.
  • 2026-07-10: Implemented first stabilization pass (CI re-enabled non-blocking, JSON reporting, health-check webServer parity, waitForTimeout removal). Remaining gap: local CMS smoke still timed out in test/e2e/auth.setup.ts for tenant-admin/author role bootstrap.
  • 2026-07-10: Focused hardening pass completed for setup auth. Replaced UI-form login in test/e2e/auth.setup.ts with Auth.js credentials API bootstrap (CSRF + callback), persisted role storage states directly, and validated with targeted Playwright smoke (auth.setup.ts, admin-rbac.spec.ts, api-v1/auth.spec.ts).