Rendered from specs/S18-mutation-score.md — edit it there, not here.

estimate8 ptsL16–32 hStryker JSON report reducer plus the committed artifact, its docs-site join and the not-measured fallback — sized by its band, no adjustment. Advisory only — estimated by the project's AI coding agent (Claude Code) on 2026-07-26 from the implementation (2109 LOC across 4 files); not part of the maturity score. Rubric: S13.

S18 — Test Strength (Mutation Score Publication)

Overview

Line coverage is already gated at 80% on all four c8 axes, and every spec clears it — the population sits between 90% and 99%. That is the gate working, but it also means coverage has no dynamic range left: cross-plotted against customer value across the scored feature specs it returns r ≈ 0.08, so it cannot rank one spec's test strength against another's and cannot answer "which valuable feature is thinly tested?".

Mutation testing can. Stryker is already configured (stryker.config.json, constitution Article V) with a break threshold, and npm run test:mutation already runs — but its output is an HTML report that is never committed, so no other tool can read it and no trend or comparison exists. This spec publishes the score: a committed, machine-readable per-file artifact generated from Stryker's own JSON reporter, which S10 then charts against the S16 value estimate.

The maturity scorer deliberately checks only that the mutation gate exists (mutation-gate, 2 points) rather than what it scores, because a full run is too expensive to make a scoring input. This spec does not change that.

User Stories

• As a maintainer, I want to see which of my valuable features have weak tests, which line coverage can no longer tell me.
• As a contributor, I want the mutation score of a file I am about to change without waiting for a full mutation run.
• As the project, I want the score in a committed, tool-neutral file so any agent, script, or docs page can read it without re-running Stryker.

Functional Requirements

• S18-SR-01 A committed artifact mutation-score.json MUST record the mutation results per source file: the mutant tallies by status and the score derived from them, plus a repository-wide total. It MUST be generated by node scripts/mutation-score.mjs from Stryker's JSON report and MUST carry a $comment marking it generated — never hand-edited, exactly like maturity-score.json, dora.json and bundle-size.json.
• S18-SR-02 The artifact MUST store the tallies, not only a percentage, so the score is auditable and recomputable by a consumer that disagrees with the formula. The formula is the mutation-testing standard — (killed + timeout) / (killed + timeout + survived + noCoverage) — and MUST be declared once in the generator script, which is its machine-readable authority; this spec's prose mirrors it.
• S18-SR-03 The artifact MUST carry provenance: the date generated and the Stryker break threshold in force at generation time. Because the score is a snapshot of code that keeps changing, any surface displaying it MUST show that date, so a stale figure is visibly stale rather than silently wrong.
• S18-SR-04 Generating the artifact MUST NOT be part of the mandatory local gate (npm run verify) or of a blocking CI job: a full mutation run takes orders of magnitude longer than the rest of the gate. It is refreshed on demand like npm run maturity, and its absence MUST degrade gracefully — consumers MUST treat a missing artifact as "not measured" and MUST NOT fail or substitute a zero.
• S18-SR-05 Files Stryker is configured not to mutate (the mutate!src/… exclusions, kept in lockstep with the coverage exclusions by npm run check:consistency) MUST be absent from the artifact rather than recorded with a zero or null score, so "not mutated by policy" is never read as "badly tested".

Non-Functional Requirements

• S18-NFR-01 The generator MUST be plain Node with no third-party dependencies, like the repository's other checkers, and the artifact MUST be tool-neutral JSON (S15).
• S18-NFR-02 The mutation score MUST NOT influence the maturity score: scripts/maturity-score.mjs MUST NOT read mutation-score.json. The scorer checks that the gate exists, and adding a score-derived input would make the maturity number depend on an artifact that is refreshed on demand and may be arbitrarily stale.

Out of Scope

• Running mutation testing in CI, or gating a pull request on the score — the break threshold already fails a local run below 60, which is the gate.
• A time series of mutation scores; the artifact is a current snapshot, and trend reporting belongs to the maturity history if it is ever wanted.
• Per-requirement mutation scores. Stryker reports per file; the file → spec join comes from the traceability matrix's impl paths, and a finer attribution would need a mapping the repository does not have.

Acceptance Criteria

1. npm run test:mutation writes reports/mutation/mutation.json, and node scripts/mutation-score.mjs turns it into mutation-score.json with a per-file breakdown and an overall score.
2. Re-running the generator on an unchanged report reproduces the file byte for byte.
3. Every file listed carries tallies whose recomputation reproduces the stored score; no file excluded by stryker.config.json's mutate list appears.
4. Deleting mutation-score.json leaves npm run verify passing and the docs site building, with the test-strength chart showing its "not measured" state.
5. scripts/maturity-score.mjs contains no reference to mutation-score.json, and npm run maturity output is unchanged whether or not the file exists.

History

• 2026-07-25 — Created after plotting per-spec line coverage against customer value across the scored features returned r ≈ 0.08, with every spec between 90% and 99%: the 80% gate compresses the population so far that coverage cannot discriminate test strength. Mutation data was already being produced by every npm run test:mutation run and thrown away with the HTML report.

Requirement traceability

Status of each requirement in specs/traceability.json. Test coverage is auto-discovered from [ID] tags in test titles and is not listed here.

Requirement	Status	Implementation	Note
S18-NFR-01	implemented	scripts/mutation-score.mjs	plain Node, no dependencies
S18-NFR-02	implemented	scripts/maturity-score.mjs	maturity scorer never reads mutation-score.json; it checks only that the gate exists
S18-SR-01	implemented	scripts/mutation-score.mjs	mutation-score.json generated from Stryker JSON report
S18-SR-02	implemented	scripts/mutation-score.mjs	tallies stored alongside the derived score; formula declared once in the script
S18-SR-03	implemented	scripts/mutation-score.mjs, docs/.vitepress/theme/SpecInsights.vue	generatedAt + break threshold recorded and shown beside the chart
S18-SR-04	implemented	scripts/mutation-score.mjs, docs/.vitepress/specs.data.ts	absent artifact degrades to not-measured; never in npm run verify
S18-SR-05	implemented	scripts/mutation-score.mjs	files stryker does not mutate are absent, not zeroed

Documented in

Documentation sections tagged spec:S18 (or one of its requirement ids) — 165 of ~391 expected words (42%). The expectation comes from this spec's evaluated complexity of 9.8 (7 documentable requirements — 5 SR, 2 NFR — weighted by kind and by each definition's length).

• Recording what broke/specs/insights · 165 words

Changes over time

• c0156392026-07-26feat(docs): add requirement lifecycle stamps and publish the mutation score