Suite Health

Run 422 · 533ea239 on main · Tue, 27 Oct 2026 00:05:49 GMT

Current run

Functional E2E · Chromium 22 23%
Functional E2E · WebKit 22 23%
Visual Regression · Chromium 33 34%
API · REST 19 20%

Suite	Cases	Pass rate	Flaky	Flaky rate last 30 runs	p50	p95	Wall clock
Functional E2E · Chromium	22	100%	0	0.61%	2.5s	7.5s	74.4s
Functional E2E · WebKit	22	100%	0	0.15%	7s	11.6s	154.1s
Visual Regression · Chromium	33	100%	1	0.5%	1.8s	7.9s	121.1s
API · REST	19	100%	0	0.18%	0.5s	1.2s	5.7s

Last 30 runs

Suite	Pass rate, 0 to 100	p95 duration
Functional E2E · Chromium
Functional E2E · WebKit
Visual Regression · Chromium
API · REST

Slowest cases in this run

Case	Suite	Duration
VR-08: Subscription widget, subscribed	Visual Regression · Chromium	32.5s
TC-11: A new visitor completes signup end to end and lands signed in	Functional E2E · WebKit	12.1s
TC-22: The confirmation page confirms the order, downloads its invoice, and Continue returns home	Functional E2E · WebKit	11.6s
TC-05: The main navigation reaches every page it links to	Functional E2E · WebKit	11.2s
TC-21: The payment form collects card details and completes the order	Functional E2E · WebKit	10.7s

Repeat offenders

Case	Suite	Runs affected
TC-08: A registered user can sign in with valid credentials	Functional E2E · Chromium	3 of 30
VR-30: Payment form, default	Visual Regression · Chromium	2 of 30
VR-08: Subscription widget, subscribed	Visual Regression · Chromium	2 of 30
TC-21: The payment form collects card details and completes the order	Functional E2E · Chromium	1 of 30
API-07: Looking up a user by email returns their profile	API · REST	1 of 30

How a failure gets classified

The rules this suite carries, how many results of this run each of them caught, and how they have landed across the window. One run says an assertion failed; the window says whether this suite argues with the application or with the host it shares. The rules stand whether or not anything matched them; the counts are the ones the Categories tab of the report shows.

Category	What it means	In this run	Last 30 runs
Visual difference	A screenshot no longer matches its baseline. Either the interface changed or the baseline is stale.	none in this run	none
Missing baseline	No committed baseline for this capture on this platform. The visual job fails rather than writing one and reporting success.	none in this run	none
Application defect	Cases parked with test.fixme() against a defect in the application under test rather than in the suite.	none in this run	none
Host unavailable	The public demo host shed the request. Infrastructure noise, not a product regression.	none in this run	none
Unexpected response shape	The endpoint answered JSON, but without the numeric responseCode every endpoint under /api carries. The answer changed shape rather than the test being wrong, which is why it is named here instead of surfacing as an assertion on an undefined field.	none in this run	none
Page did not load	Failed inside a page object's navigation, before any of the test's own steps: the page the test asked for never arrived. In practice this is the public demo host shedding load or serving an empty page; a landing heading the application changed would land here too, which is why the name says what happened rather than guessing the cause.	none in this run	none
Timed out waiting for state	The application never reached the state the case was waiting for, within the timeout.	none in this run	1 in 1 of 30 runs
Assertion failed	An assertion about the application's behaviour did not hold. The most interesting category here.	none in this run	1 in 1 of 30 runs
Flaky	Passed only after an earlier attempt of the same case failed. The retry kept the run green; the case still needs its cause found.	1	3 in 3 of 30 runs

Visual baselines

What a baseline regeneration would have to review, and when each area's captures were last agreed with. A browser upgrade moves glyph rendering and invalidates all of them at once, which is why the number and the date are worth seeing before one is taken.

Area	Baselines	Last changed
cart	3	2026-10-27
checkout	2	2026-10-27
confirmation	1	2026-10-27
home	10	2026-10-27
login	2	2026-10-27
payment	3	2026-10-27
product-detail	4	2026-10-27
products	3	2026-10-27
signup	5	2026-10-27
All areas	33

Thresholds

Pass rate: 90% or above is healthy, 85 to 90% is acceptable during active development, below 85% means the suite has a stability problem rather than the application.
Flaky rate: Measured across the last 30 runs, not one. Below 1% is the target. Past 5% the suite stops being believed, and a suite nobody believes is worse than no suite.
Duration: Tracked as a trend, not a fixed limit. What matters is whether it is growing faster than coverage.
Coverage: New coverage exists only because a genuine state on the page or the API was found missing and verified live before it was added, never to pad the count. The WebKit run is not a coverage number of its own: it replays the same functional cases on a second engine.