fix: skip PlaywrightCheck constructs during pw-test [RED-155]

[miliberlin]

Background

Reported internally by Stefan: when a project has a PlaywrightCheck construct defined in a .check.ts file, pw-test incorrectly includes it alongside the vanilla Playwright tests. This causes pw-test to run two checks — the construct-based check and the ad-hoc one — when it should only run the ad-hoc one:

__checks__/critical-suite.check.ts
  ✔ Critical User Flows (8s)
playwright.config.ts
  ✖ Playwright Test:  (11s)

The construct-based check passes (self-contained config/deps), but the ad-hoc playwright.config.ts check fails because the construct's bundling context clobbers its dependency resolution.

Root cause

parseProject() was not scoped to the pw-test command:

1. loadAllCheckFiles() ran for all commands including pw-test, picking up .check.ts files that contain new PlaywrightCheck(...) constructs
2. checklyConfigConstructs (from config file evaluation) were added to the project unfiltered, even for pw-test

This meant pw-test would end up with extra PlaywrightCheck instances in the project. Since the checkFilter keeps all PlaywrightCheck instances, both the construct-based check and the ad-hoc check made it into the bundle. The construct's dependency tree interfered with the ad-hoc check's bundling.

Changes

Two changes in parseProject() when currentCommand === 'pw-test':

1. Filter PlaywrightCheck constructs from checklyConfigConstructs before adding to the project
2. Skip loadAllCheckFiles() entirely

Before/after (released CLI v7.0.1 vs this fix)

Before:

Running 2 checks in eu-central-1.

playwright.config.ts
  ✔ Playwright Test (15s)
src/__checks__/synthetics/07-pw-check-construct.check.ts
  ✔ File Based PW Check (should be ignored by pw-test) (13s)

2 passed, 2 total

After:

Running 1 checks in eu-central-1.

playwright.config.ts
  ✔ Playwright Test:  (15s)

1 passed, 1 total

Test results

Each test project includes a PlaywrightCheck construct in a .check.ts file, a local project dependency (src/utils/selectors.ts), and an npm devDependency (lodash). The test files import both to verify dependency resolution works correctly.

Issue	Scenario	Before (v7.0.1)	After (fix)
RED-122	PlaywrightCheck in .check.ts file	2 checks run	1 check run
RED-119	PW construct causes config to be ignored	2 checks run	1 check run
RED-118	PW construct blocks dependency resolution	2 checks run	1 check run

All three child issues verified fixed. The released CLI (v7.0.1) consistently runs 2 checks (the bug), while the fix correctly runs only 1 ad-hoc check. Both local and npm dependencies resolve correctly with the fix.

Before sessions: RED-122 · RED-119 · RED-118
After sessions: RED-122 · RED-119 · RED-118

🤖 Generated with Claude Code

[miliberlin]

Summary of why we need to tweak the config for this test below. I think this is correct since this is how a user would set up the project, but please double-check.

Before our fix, the webServer test worked like this:

1. parseProject() runs with --emulate-pw-test
2. loadAllCheckFiles() picks up test.check.ts, which calls new PlaywrightCheck(...) with playwrightConfigPath: './playwright.config.ts'
3. The PlaywrightCheck constructor runs validateWebServerConfig(), sees webServer in the playwright config, and emits the warning

Our fix adds if (currentCommand !== 'pw-test') around loadAllCheckFiles(), so step 2 never happens — test.check.ts is never loaded, no PlaywrightCheck is created, no warning fires.

But there's a second code path that creates PlaywrightCheck instances: loadPlaywrightChecks(). This function still runs for pw-test. It creates an "ad-hoc" PlaywrightCheck when playwrightConfigPath is set in the checkly config — this is the same path that real users hit when they have checks: { playwrightConfigPath: './playwright.config.ts' } in their checkly.config.ts.

By adding playwrightConfigPath to the fixture's config, the ad-hoc check gets created via loadPlaywrightChecks() instead of via loadAllCheckFiles(). The webServer validation fires on this ad-hoc check, and the test passes.

This also makes the test more realistic — in practice, pw-test users always have playwrightConfigPath in their config (that's how pw-test knows which playwright config to use). The old fixture relied on a .check.ts file to create the PlaywrightCheck, which is exactly the scenario our fix is designed to skip.

[miliberlin]

@ferrandiaz would you have time to check this today? We'd like to close the Polish pw-test project soonish.