Suite health

Run 146 · 9fd0e0a1 on main · Mon, 14 Sep 2026 02:52:59 GMT

Current run

functional (Chromium) 20 33%
functional (WebKit) 20 33%
visual (Chromium) 20 33%

Suite	Cases, cap	Pass rate	Flaky rate	p50	p95	Wall clock
functional (Chromium)	20 / 20	100%	0%	4.9s	8.1s	112.9s
functional (WebKit)	20	100%	0%	5.8s	12.5s	155.9s
visual (Chromium)	20 / 20	100%	0%	4s	6.6s	89.1s

Last 27 runs

Suite	Pass rate, 0 to 100	p95 duration
functional (Chromium)
functional (WebKit)
visual (Chromium)

Slowest cases in this run

Case	Suite	Duration
TC-17: a signed-in user can complete an order end to end	functional (WebKit)	15.4s
TC-04: signing up with an already registered email is rejected	functional (WebKit)	12.5s
TC-02: a registered user can sign in with valid credentials	functional (WebKit)	12.1s
TC-06: logging out returns the visitor to the anonymous state	functional (WebKit)	11.6s
TC-01: a new visitor can register and lands in the signed-in state	functional (WebKit)	11.3s

Repeat offenders

Case	Suite	Runs affected
TC-16: an anonymous visitor cannot reach checkout	functional	3 of 27
TC-17: a signed-in user can complete an order end to end	functional	3 of 27
VR-04: catalog grid	visual	3 of 27
TC-05: a signed-in user can delete their account	functional	3 of 27
VR-19: delivery address block	visual	2 of 27

How a failure gets classified

The rules this suite carries, and how many results of this run each of them caught. The rules stand whether or not anything matched them; the counts are the ones the Categories tab of the report shows.

Category	What it means	In this run
Visual difference	A screenshot no longer matches its baseline. Either the interface changed or the baseline is stale.	none in this run
Missing baseline	No committed baseline for this capture on this platform. The visual job fails rather than writing one and reporting success.	none in this run
Application defect	Cases parked with test.fixme() against a defect in the application under test rather than in the suite.	none in this run
Host unavailable	The public demo host shed the request. Infrastructure noise, not a product regression.	none in this run
Timed out waiting for state	The application never reached the state the case was waiting for, within the timeout.	none in this run
Assertion failed	An assertion about the application's behaviour did not hold. The most interesting category of the six.	none in this run

Thresholds

Pass rate: 90% or above is healthy, 85 to 90% is acceptable during active development, below 85% means the suite has a stability problem rather than the application.
Flaky rate: Below 1% is the target. Past 5% the suite stops being believed, and a suite nobody believes is worse than no suite.
Duration: Tracked as a trend, not a fixed limit. What matters is whether it is growing faster than coverage.
Coverage cap: Twenty cases for the functional suite and twenty for the visual one. New coverage replaces an existing case rather than adding to the count. The WebKit run has no cap of its own: it replays the same functional cases on a second engine.