Rendered from specs/S08-e2e-testing.md — edit it there, not here.

estimate21 ptsXXL80 h+INFRAReal-VS-Code E2E: harness download, xvfb, CI wiring, flake management — infrastructure effort invisible in LOC. Re-estimated after release-scoped demo selection (S08-SR-12..14: demo<->spec registry, change-detection script, refresh-gifs.yml rewiring) moved the live LOC into the next band. Advisory only — estimated by the project's AI coding agent (Claude Code) on 2026-07-25 from the implementation (2656 LOC across 18 files); not part of the maturity score. Rubric: S13.

S08 — End-to-End Testing

Overview

The unit suite runs against a mocked vscode module. The mock's getConfiguration().inspect() returns the same value for every settings layer and ignores resource scoping entirely, so an entire class of real bug — wrong ConfigurationTarget, folder-scoped settings in multi-root workspaces, path-prefix mismatches between what a binding writes and what a reader matches — passes the unit suite by construction. The multi-root binding bugs found in mid-2026 (F04-FR-13 and the findBoundSchemaPath scope/prefix mismatches) are exactly this class. The Playwright scripts under src/test/e2e/ do not cover it either: they are demo-GIF recordings (screenshots, no assertions) that only run in the GIF-refresh workflow.

This spec adds a real end-to-end suite that runs the packaged extension in an actual VS Code instance (@vscode/test-electron), against real settings files, real multi-root workspaces, and the real filesystem — the layer where the mock necessarily lies.

Requirements

Harness

• S08-SR-01 An E2E suite MUST run the extension in a downloaded, real VS Code build via @vscode/test-electron (npm run test:integration), driving it through the vscode API from an extension-host test runner (no UI scripting required). It MUST be runnable headless in CI (xvfb).
• S08-SR-02 Fixtures MUST include at least: a single-folder workspace, a multi-root workspace (≥ 2 folders with distinct .vscode/settings.json files), and data/schema files in JSON, YAML, and TOML.
• S08-SR-03 E2E tests MUST use the same mocha tdd interface and requirement-tag convention as unit tests; a [ID] tag in an E2E test title MUST count as test coverage in npm run check:traceability, so manual requirements verified by an E2E test can be promoted to implemented.

Mandatory Scenarios

• S08-SR-04 Settings-scope round-trips MUST be covered as a matrix: for each of Global / Workspace / WorkspaceFolder × JSON / YAML, bind a schema, then assert (a) the entry landed in the correct settings file with the correct path form, and (b) the extension's own readers (findBoundSchemaPath → status bar, validator) see the binding — in both single-folder and multi-root fixtures, including the cross-folder case (schema in folder A, data file in folder B).
• S08-SR-05 Inline-binding round-trips MUST be covered for JSON, YAML, and TOML: bind, assert the document text, validate, remove, assert clean — including a TOML file whose first content is a [table] header and a schema path containing characters that require escaping in the target syntax.
• S08-SR-06 One smoke test per user-facing command in package.json#contributes.commands MUST at minimum execute the command against a fixture and assert it neither throws nor leaves an error notification.
• S08-SR-07 The S03 startup/perf budgets (e.g. S03-SR-13's p95 command latency) MUST be measured in the E2E run and reported, so planned performance requirements can be promoted with real numbers.

CI Integration

• S08-SR-08 The E2E suite MUST run in CI on every PR (own job, may be slower than unit tests). It MAY start non-blocking while flakiness is measured, but MUST be promoted to a required check once it has run clean for 2 consecutive weeks; the promotion MUST be recorded in this spec's History.

  Non-blocking MUST mean "not a required status check", never continue-on-error: true. The two are not interchangeable: branch protection decides whether a red job blocks the merge button, while continue-on-error makes the job report green when it failed. The second does not just hide the failure — it destroys the evidence this requirement depends on, because a job that always reports success can never be observed running "clean for 2 consecutive weeks", so the promotion criterion can never be evaluated and the job stays provisional forever.

• S08-SR-09 The suite MUST run on both Linux and Windows runners — path-separator and drive-letter handling (absolute paths embedded by F04-FR-13/F10-FR-05, TOML string escaping) are platform bugs the unit suite cannot catch on Linux alone.

UI smoke (Playwright demo scripts)

The src/test/e2e/ Playwright scripts capture screenshot frames for the demo GIFs and contain no assertions, so historically they ran only in the GIF-refresh workflow. They come in two variants per feature: a mouse twin (demo-<x>-mouse, animated cursor — the frames the GIF pipeline consumes) and a command-palette twin (demo-<x>, no mouse — whose frames the GIF pipeline does not consume). The command-palette twins are otherwise wasted effort, yet they exercise the real feature flow through the actual VS Code UI, so they have value as a lightweight UI smoke signal the assertion-based integration suite (API-level) does not provide.

demo-showcase-mouse is the one exception to "capture screenshot frames": it records the real X11 display instead (see the 2026-07 History note) and feeds scripts/make-showcase-gif.mjs, not scripts/make-gifs.mjs. It still follows the mouse/non-mouse split below — mouse in the title is what routes it to the GIF-refresh job rather than the smoke job, regardless of capture mechanism.

• S08-SR-10 The command-palette (non-mouse) demo scripts MUST run as a UI-smoke job (Playwright launching real VS Code, headless via xvfb), passing when each demo flow completes without throwing or timing out — i.e. crash / broken-selector detection, not pixel assertions. It runs at release time (the GIF-refresh workflow, on: workflow_run after Release Please), not on every PR/push — see the History note on why — as a job parallel to (not dependent on) the GIF-refresh job, so neither lengthens the other's critical path. It MUST start non-blocking while flakiness is measured, in the same sense S08-SR-08 defines: excluded from the required status checks, and never via continue-on-error: true. It runs after a release rather than on a pull request, so nothing about it can block a merge in the first place — swallowing its result would buy nothing and cost the only signal it exists to produce.
• S08-SR-11 The mouse demo scripts MUST NOT run in the smoke job, and the GIF-refresh workflow's refresh-gifs job MUST run only the mouse scripts (the non-mouse frames it never consumes) — so each Playwright variant runs in exactly one place: mouse → GIFs, non-mouse → smoke. The split is by test-title grep on mouse, which every mouse test title contains and no non-mouse title does.

Release-Scoped Demo Selection

• S08-SR-12 At release time, the GIF-refresh workflow's refresh-gifs job and the UI-smoke job (e2e-smoke) MUST determine which specs changed or were newly added since the previous release tag, and MUST run demo Playwright scripts (both the mouse and command-palette twins) and regenerate GIFs only for the demos mapped to those specs, instead of the full set on every release. When no previous release tag is resolvable (the first release, or a checkout too shallow to reach one) the run MUST default to every demo, mirroring S09-SR-02's "no usable base → run everything" fallback.
• S08-SR-13 The demo ↔ spec mapping MUST be declared in exactly one place (scripts/demo-registry.mjs), consumed by both the GIF generator (scripts/make-gifs.mjs) and the change-detection script (scripts/detect-changed-features.mjs) — adding a demo for a new feature, or reassigning one to a different spec, is a one-file edit. demo-showcase/ demo-showcase-mouse is mapped like any other entry, to every spec its narrative touches (F01, F02, F03, F04, F06, F09, F10, F18 per the History notes above), even though its GIF is produced by a different script (scripts/make-showcase-gif.mjs) than the other demos.
• S08-SR-14 This selection applies only to the demo/GIF pipeline (refresh-gifs.yml's two jobs). The assertion-based integration suite (npm run test:integration, S08-SR-08) is unaffected and continues to run in full on every PR — it is not release-scoped and not keyed to which specs changed, since it is regression coverage for the whole extension, not a per-feature demo capture.
• S08-SR-15 The refresh-gifs.yml workflow_dispatch trigger MUST accept an optional demos input — a comma-separated list of demo names (as they appear in scripts/demo-registry.mjs), or the literal all — that overrides S08-SR-12's automatic spec-diff selection for that run only. This covers the case S08-SR-12 cannot: a change to a demo's script (the Playwright test itself, or a doc/History note) rather than to one of its mapped spec files, which produces no spec-diff signal for detect-changed-features.mjs to key off, yet still needs the GIF/smoke run it would otherwise only get by coincidentally touching a mapped spec too. An unrecognised demo name in the override MUST fail the run rather than silently produce an empty or partial selection. Leaving the input empty (the default) MUST fall through to S08-SR-12's normal behaviour unchanged.

GIF Encoding

• S08-SR-16 Every demo GIF — the frame-stitched ones and the showcase recording alike — MUST be encoded with ffmpeg's two-pass palette pipeline (palettegen=stats_mode=diff then paletteuse=…:diff_mode=rectangle). One encoder for all of them: a per-frame palette optimised for a mostly-static editor window, re-encoding only the region that changed between frames.
• S08-SR-17 The GIF pipeline MUST NOT depend on a package requiring a native build. canvas compiles against cairo/pango at install time, which is why a plain npm ci fails in a minimal container and why the repository needs a bootstrap script at all — a cost paid on every clone by everyone, to serve a step that runs only at release time on one Linux runner that already has ffmpeg installed.
• S08-SR-18 Encoding MUST preserve the captured frames' own resolution. Downscaling to save bytes is unnecessary: at identical dimensions the palette pipeline is roughly ten times smaller than the octree quantiser it replaces, so there is nothing to trade legibility for.

Demo content and placement

• S08-SR-19 Every demo MUST show its feature succeeding. A capture that ends on a refusal or a no-op is a defect in the demo script, not a documentation gap: the script MUST seed whatever state the feature needs (a bound schema, a fixture data file) before performing the action it demonstrates. demo-validation shipped for months ending on "No schema bound to person-invalid.json. Bind one first." — it opened an unbound file, so the one demo of the extension's headline feature never validated anything.

  Succeeding is not enough on its own: the surface the step is about MUST be on screen while it changes. A step whose subject is the rendered preview refreshing has to have the preview visible in the captured region — the command running correctly off-screen demonstrates nothing to a reader. demo-showcase shipped this way too: it closed the schema tab for "a beat with only the viewer on screen", which made VS Code drop the now-empty editor group and reopen the schema inside the preview's group, hiding the preview behind it. The panel was visible for 7 of 92 seconds, and neither the live-title-update step (F02) nor the configure step (F09) — both of which exist purely to show the preview changing — had it on screen at all. Where a demo can assert this, it SHOULD: a capture that silently records the wrong thing is the failure mode this requirement exists to catch, and nothing else in the pipeline notices it.

• S08-SR-20 Every entry in scripts/demo-registry.mjs MUST be embedded on the docs site in both places a reader looks: the landing page's demo gallery (docs/.vitepress/theme/QuickDemo.vue) and the guide section that documents the command it shows. A GIF regenerated at release time and never displayed is pure payload. The README is deliberately the exception: it embeds demo-showcase.gif alone, because the marketplace renders it and seventeen inline GIFs would make that page unusable.

• S08-SR-21 A frame-stitched demo SHOULD stay under 30 seconds of playback. Past that a reader scrubs rather than watches, and the file grows without teaching more. The 25 frame-stitched demos currently run 5.4–19.9 s, so the budget is headroom rather than a diet; it exists to keep a demo from quietly growing past the point where anyone watches it to the end. demo-showcase is the deliberate exception — it is the one end-to-end narrative — but it is held to the same principle: it MUST tell one story, not several chained together. It ran 92 s carrying two acts (author a schema and preview it; then validate and bind a different file); the second was cut because every feature in it already had a focused demo, and a tour that changes subject halfway is two demos in one file. That took it to 40 s — still over the 30 s budget, but close enough that the remaining gap is pacing rather than scope. Shortening it means re-recording, which needs a real X11 session; it cannot be done by re-encoding.

Harness notes (implementation)

• The toml language id and the yaml.schemas configuration key are not provided by any bundled VS Code extension (they normally come from the marketplace extensions tamasfe.even-better-toml and redhat.vscode-yaml). A contribution-only fixture extension (src/test/integration/fixtures/test-lang-support) declares both and is loaded via extensionDevelopmentPath — which is honoured even under --disable-extensions — so .toml files get a real language id and folder/workspace/user-scoped yaml.schemas writes succeed, all without a marketplace --install-extension step (satisfying S08-NFR-02 and avoiding the Windows .cmd spawn pitfalls that step brings). Because redhat.vscode-yaml itself is absent, inline YAML binding writes the plain $schema: key form rather than the # yaml-language-server: directive (F10-FR-09).
• Fixture schemas use draft-07 (http://json-schema.org/draft-07/schema#): the extension's validator uses Ajv's default export, which bundles draft-07 and does not know the draft 2020-12 meta-schema. This is also why the validator must parse YAML-format schema files (F03-FR-14) — the schema.yaml fixture proved loadSchema was JSON-only, an inconsistency with the other schema consumers that S08 was built to surface.

Non-Functional Requirements

• S08-NFR-01 The E2E job SHOULD finish in under 10 minutes; scenario count is bounded by the matrix above, not by porting the unit suite.
• S08-NFR-02 E2E tests MUST NOT hit the network: remote-schema scenarios use a local HTTP fixture server, keeping runs deterministic and offline (S04, S05).

History (encoding)

• 2026-09-02 — The 16 frame-stitched demos moved from gif-encoder-2's octree quantiser to the same ffmpeg palette pipeline make-showcase-gif.mjs already used, and canvas/gif-encoder-2 were dropped (S08-SR-16/17/18).

  The earlier History note reasoned that the screenshot pipeline was "simpler and has no external-binary dependency beyond the already-required canvas". Both halves of that turned out backwards. ffmpeg is not an additional dependency — the refresh workflow already installs it for the showcase — and canvas is not free: it is a native build, the reason npm ci fails in a minimal container, and the reason scripts/bootstrap.mjs exists.

  The measured result on the largest demo, at unchanged dimensions: 4.03 MB → 0.39 MB, visually indistinguishable. Across the 16 frame-stitched GIFs the committed payload drops from ~28 MB to ~3.4 MB. Re-encoding needs neither X11 nor the original frames — decoding each committed GIF and re-encoding it produces the same timing — so the existing GIFs were re-encoded in place rather than waiting for the next release to regenerate them. demo-showcase.gif was left alone: it already came out of this pipeline, so a second lossy pass would buy ~4% for a generation of quality. It stays the outlier at 7.8 MB — a length problem, not an encoding one (S08-SR-19).

  What still needs a real recording session — and so is not addressed here — is trimming demo-showcase (S08-SR-21).

• 2026-09-03 (showcase composition) — demo-showcase was recorded with the preview panel on screen for 7 of its 92 seconds. Closing the schema tab to get "a beat with only the viewer" removed the editor group holding it, and the reopened schema editor then landed inside the preview's own group as a new tab, hiding it for the remainder of the run. The two steps that exist to show the preview reacting — the live title edit and the render-template change — both played against a full-width editor.

  The schema tab is no longer closed: editor left, docs right, for the whole narrative, which is also what using the extension actually looks like. That removed the need to reopen the file at all, and with it the Explorer/Quick Open flakiness the Ctrl+Shift+T workaround existed to dodge. expectPreviewVisible() now asserts the panel at each of those moments, so a capture that records the wrong thing fails instead of shipping (S08-SR-19).

  The first re-record attempt failed before reaching any of that, on pre-existing code: step 3 aimed the cursor straight at the .codicon-close icon of the background person-valid.json tab. VS Code reveals that icon only on the active or hovered tab, and glideToLocator waits for visibility before moving the pointer — so it waited for the very thing only the move would produce. That had resolved by luck for a long time and stopped on VS Code 1.136.1. The demo now glides to the tab, clicks it (which reveals its close button), and falls back to the keyboard shortcut. Worth recording because the class generalises: a mouse demo must not target a control that only appears on hover, since the wait that precedes the move cannot be satisfied by the move itself.

  The second attempt then failed on the new assertion itself, which is the guard behaving correctly about the wrong thing: it checked instantly, before the multi-second beat that follows each Preview click, and preview rendering shells out to a Python renderer. The instant check had been a deliberate choice — "the panel should already be up, so waiting would mask a slow reveal" — which is simply untrue at a site that has just created the panel. It now waits, bounded, and each call sits after its beat, so what it asserts is the state the recording actually contains.

  The third attempt reached 2.2 of the run's ~4 minutes — past all three preview assertions, so the layout fix itself held — and died at step 12, waiting for "Generate Types from This Schema" in the More Actions menu. That command had moved: editor/title now contributes the jsonschema.schemaMenu submenu, and every non-icon command sits one level down inside it. The restructuring landed earlier on the same branch, and nothing caught it, because the two attempts above both died at step 3 — this was the first full pass over the current package.json since. The general rule: restructuring a menu contribution invalidates every demo that clicks through that menu, and a demo only proves that when it runs end to end. A demo failing early hides every later step's breakage, so a fix that gets a demo further is not evidence that the rest still works.

• 2026-09-04 (showcase scope) — With the preview finally on screen where it belonged, the recording's real problem became legible: it was two demos. Act one authored a schema and previewed it; act two switched to a different file to validate, inline-bind, re-validate and trigger IntelliSense, then navigated back. That second act was ~45 of the 91 seconds, ran with a full-width editor and no preview throughout, and every feature in it (F03 validate, F04 bind) already had its own focused demo on the docs site.

  Cut to one story: infer → preview → live-edit → configure → generate code. The schema editor and preview stay side by side from the moment the preview opens until the end, so there is no longer any stretch of the tour where the headline surface is absent.

  One real cost, recorded rather than glossed: F10 (inline $schema binding) loses its only demo — it was covered nowhere but that middle act. It joins the backlog in the entry below at an S16 value of 13.2, which places it third, above every remaining gap except F34 (18) and F12 (15.36) — so it is not a quiet demotion. Buried mid-tour it was not discoverable anyway: a reader looking for "how do I bind a schema inline" would not have found it 60 seconds into a video about authoring one.

  The per-step pauses were also cut. They had been fixed waitForTimeout values sized for a worst-case Python render; expectPreviewRendered() now waits for the rendered content itself (the Required badge, emitted by both templates the demo uses — verified by running json-schema-for-humans against this fixture under each, not read off a GIF frame), so render time is absorbed adaptively and the surrounding beats exist only for the reader.

• 2026-09-04 (closing the demo gap) — Nine demos written, seven kept: F34, F10, F12, F30, F15, F24 and F23. F08 and F26 were removed after recording proved them undemoable in this harness (below). F25 was closed without a new GIF: the existing quick-fix demo's fixture types "payed" against an enum containing "paid", so the ranked suggestion is F25 on screen, and a second recording of the same lightbulb would teach nothing. It was added to that demo's specs instead.

  Three needed infrastructure that did not exist, all of it small and reusable:

  • helpers/fixtureServer.ts serves fixtures over loopback HTTP, the mechanism S08-NFR-02 already names. F12 fetches its catalog with fetch(), which in the extension host is undici and does not support the file: scheme — so it cannot be demonstrated against a path on disk, however local it is.
  • F15 diffs against a workspace file, not Git HEAD: the "Git HEAD" baseline row never appears under --disable-extensions (see F26 below). The baseline is picked through a native dialog stubbed in the main process.
  • runDemo now passes the ElectronApplication and the launch's workspaceDir to the demo body, for the native-dialog stubbing demo-showcase-mouse already did by hand — which is what makes the F15 dialog stub above possible.

  A latent break found while doing this: clickEditorOverflowAction still clicked the overflow menu's top level, which F34's submenu grouping had emptied of commands months earlier. Four shipped demos (bundling, codegen, migrate, sample-data) go through that helper and would all have failed on their next re-record — the same wall demo-showcase hit. They passed in the meantime only because none of them was re-recorded. The helper now traverses the submenu, and does so conditionally, so flattening the menu again would not silently break it a second time.

  Two of the nine did not survive recording, and both taught the same lesson. They were written, wired and merged before anything ran them, then removed once CI showed what they actually did:

  • F26 (compatibility CodeLens) — the lens reads Git HEAD through vscode.extensions.getExtension('vscode.git'), and the harness launches with --disable-extensions, which takes built-in extensions down as well. Seeding a real repository did not help, and neither did a launch variant that kept built-ins enabled. The harness cannot both show this lens and keep the built-in chat UI out of frame, so F26 has no demo.
  • F08 (local schema cache) — jsonschema.cacheSchemaLocally is contributed to commandPalette with "when": "false": it is deliberately unreachable from the palette, so the demo written against the palette could never have worked. Rebuilt 2026-09-09 against its real entry point, the $schema-line quick fix — see the note below.

  The F08 removal was half wrong, and the correction is worth more than the removal was. "Its real entry points ... need a failing fetch" was read off the docs, not the provider: SchemaAuthCodeActionProvider gates the caching action only on the cursor sitting on a line with a remote $schema URL. Schema-load diagnostics merely make the action preferred and give it a gutter lightbulb — they are not required for it to be offered, so a plain Ctrl+. reaches it on a perfectly healthy schema. F08 has a demo again, driven that way, with a fatal wait on the "Schema cached" confirmation: the $schema line does not change when the command succeeds, only the binding behind it, so there is nothing else on screen to prove the download happened.

  The lesson repeats the one below at one remove: the first F08 script was written from the command list, and the removal was argued from the guide page. Neither is the provider. Read the code that gates the surface.

  Both original failures were caught only because the mouse twin failed. Both plain twins "passed" while doing nothing — runCommand types a name, presses Enter on whatever the palette highlights, and returns; with the command absent, that is a no-op the demo cannot detect. F26's first recording passed the same way, with its CodeLens wait wrapped in .catch().

  The rule that generalises, and the reason S08-SR-19 exists: a demo step that cannot fail cannot be trusted. A .catch() on the wait that proves the feature rendered, or a palette entry taken on faith, converts a broken demo into a shipped GIF of nothing happening. Where a demo can assert its subject is on screen, it MUST, and that assertion MUST be fatal.

  Five gaps remain, and none of them is a matter of effort:

  • F11, F19 (TOML support and TOML IntelliSense) — every TOML when clause tests resourceLangId == toml, and this extension contributes no languages entry, so the id only exists when a marketplace TOML extension is installed. The demo harness launches with --disable-extensions (S08-NFR-02, and the Copilot focus-stealing this file documents), so a .toml file opens as plaintext and no command is offered. Demoing these means either shipping a language contribution or installing a third-party extension into the demo profile — a product decision, not a demo one.
  • F32 (AI-assisted authoring) and the language-model half of F33 need a model behind VS Code's Language Model API, which means Copilot Chat: not installed, not installable offline, and non-deterministic output even if it were — a GIF whose content changes per run is not a demo.
  • F27 (CLI) and the MCP half of F33 are terminal surfaces, not editor ones. They can be filmed in the integrated terminal, but jstk is not on PATH in the demo workspace, so the recording would open with a shim or an export PATH= line that is not how anyone uses the tool. guide/cli.md and guide/ai.md already carry the exact commands as copyable text, which is more useful than a video of someone typing them.

• 2026-09-02 (coverage audit) — Of 34 feature specs, 16 had no demo at all. Ranked by the S16 value estimate, the gaps were: F34 (18), F12 (15.36), F31 and F33 (11.52), F08 (10.56), F27 and F29 (7.68), F15 and F30 (7.04), F11, F19 and F25 (5.28), F24 (4.8), F26 (3.52), F32 (2.4), F23 (1.35).

  outline (F31) and schema-tests (F29) were added in that pass — the two with the highest value among the interactions that reuse an existing demo pattern outright. The rest remain a backlog, in that order. F12's demo needs a local catalog fixture rather than SchemaStore (S08-NFR-02 forbids network access), and F32's needs a language model, so neither is a straight copy of an existing pattern.

  New demo scripts cannot be run in a container without an X server and a VS Code download, so the two added here are unverified until the next refresh-gifs run. That is deliberately survivable: the smoke job is non-blocking, make-gifs.mjs skips a demo whose frames are missing, and the docs gallery renders a placeholder for a GIF that does not exist yet — a broken new demo costs a missing image, not a red build.

  Correction (2026-09-04): that last clause is only half true, and the half that is wrong breaks CI. The gallery (QuickDemo.vue) does degrade gracefully — it swaps in a placeholder from an @error handler at runtime. A guide-page markdown embed does not: ![…](/demo-x.gif) is resolved by Rollup at build time, so referencing a GIF that is not in docs/public/ yet fails npm run build --prefix docs outright with "Rollup failed to resolve import". Adding a demo therefore has an ordering constraint the earlier note denies: the guide embed cannot be committed before the GIF exists, or the docs job goes red and stays red until the refresh run lands the file. Either land the embed in the same change as the GIF, or add it in a follow-up.

Out of Scope

• UI pixel/screenshot assertions. The mouse demo scripts remain a screenshot-only pipeline; the non-mouse twins are promoted to a crash-level smoke signal only (S08-SR-10), not visual assertions.
• Marketplace-install/packaging tests beyond loading the compiled extension.
• Replacing unit tests — pure-logic coverage stays in the mocked suite.

Acceptance Criteria

1. npm run test:integration downloads VS Code, opens the multi-root fixture, and passes locally and in CI on Linux + Windows.
2. A deliberately reintroduced scope bug (e.g. dropping the resource argument from findBoundSchemaPath's getConfiguration) fails at least one S08-SR-04 test.
3. npm run check:traceability counts [ID] tags found in E2E test titles.
4. CI shows the E2E integration job (S08-SR-08) on PRs; its History note records when it became a required check. The UI-smoke job (S08-SR-10) shows on the "Refresh Demo GIFs" workflow run instead, per the History note below.
5. A release whose diff since the previous tag touches only specs/F06-*.md runs (and its GIF regenerates for) only demo-inference/ demo-inference-mouse and demo-showcase/demo-showcase-mouse (F06 is one of showcase's constituent specs) — every other demo is skipped in both refresh-gifs and e2e-smoke. A release with no resolvable previous tag runs every demo, unchanged from prior behavior.

Relation to Existing Specs

• Verifies F04/F10/F11 binding behaviour at the real-API layer; measures S03 budgets; complements Article V unit coverage — manual-status requirements throughout the matrix become promotable once an E2E test tags them.

History

• 2026-08 — Rewrote demo-showcase/demo-showcase-mouse's narrative from five steps to thirteen, widening its constituent specs from F01/F02/F06 to F01/F02/F03/F04/F06/F09/F10/F18 (S08-SR-13). New narrative: open a good JSON data file, generate a schema from it and save it (F06); hide the raw JSON tabs so only the rendered schema viewer shows (F01); reopen the schema and live-edit its title, watching the viewer refresh (F02); configure the viewer (via jsonschema.config, F09) to show Expand all/Collapse all, and use both buttons; close every tab and open a bad JSON example; validate it (a "no schema bound" warning, F03) and inline-bind it to the generated schema straight from the warning's own action (F04/F10); validate again to show the binding doing real work; trigger IntelliSense and Ctrl+click the inline $schema value to open the schema file (both built-in VS Code JSON language features — this extension contributes no document-link/completion provider of its own for a data file's $schema field, so nothing in this repo's own code is exercised by those two beats); generate TypeScript types from the schema (F18); and finish by flipping jsonschema.preview.autoOpen in the Settings UI and reopening the schema file to show the preview open on its own with no click. The Expand all/Collapse all buttons required switching away from the default "flat" json-schema-for-humans template (chosen specifically to avoid a CDN fetch — see F01's History) to js, which loads jQuery/ Bootstrap from a CDN. This was verified directly before writing the demo, not assumed: js's template also references two local sibling files (schema_doc.css/schema_doc.min.js) that PreviewWebPanel.ts never serves (it reads only the single generated HTML file into a string, discarding any sibling assets JSON-schema-for-humans writes alongside it), but a headless-Chromium check against the tool's actual generated output confirmed their absence only costs a cosmetic style pass and one harmless anchorOnLoad console error — the buttons themselves are driven by Bootstrap's data-toggle="collapse", wired up once jQuery/Bootstrap load from the CDN, and that still works. js_offline (the same template with every asset bundled locally, no CDN) was considered first and is the better long-term fit for this project's zero-network posture, but its local relative asset paths (css/…, js/…) have no working code path in PreviewWebPanel.ts at all — there is no <base>/asWebviewUri rewriting and no mechanism to keep a per-panel asset directory alive for the webview's lifetime, so unlike js's CDN links (which at least resolve), js_offline's local links would 404 unconditionally. Properly supporting js_offline would need webview-resource-serving work in PreviewWebPanel.ts itself — real product scope, not a demo-script change — so this rewrite deliberately took the already-working js template instead and left that as a candidate future spec rather than building it under this task. demo-showcase (command-palette twin) was extended to match — Configure Preview, Validate, inline Bind, and Generate Types all now run there too, as command-palette smoke coverage (S08-SR-10) — but continues to skip the IntelliSense/Ctrl+click beats (no command of this extension's own is behind them) and binds/generates against the pre-existing, hand-curated person.schema.json rather than the just-generated file, unchanged from its original simpler-on-purpose design (see the entry below).
• 2026-07-25 — Added S08-SR-12..14: the GIF-refresh workflow now runs demo Playwright scripts and regenerates GIFs only for the demos mapped to specs that changed or were added since the previous release tag, instead of the full 17-demo set every release. Scoped deliberately to the demo/GIF pipeline only — test:integration (S08-SR-08) is whole-extension regression coverage, not tied to a release's feature diff, and stays unscoped.
• 2026-07 — Added demo-showcase/demo-showcase-mouse, a sixth-ish demo pair that chains several features into one continuous session instead of one GIF per feature — the only demo GIF referenced from README.md, the per-feature GIFs stay docs-site-only via docs/.vitepress/theme/QuickDemo.vue. demo-showcase-mouse's content settled on: open a JSON data file from Explorer, generate a schema from it (F06), save that schema to a real path with no visible dialog (Electron's native save dialog is stubbed via app.evaluate — Preview shells a Python renderer that reads a real file from disk, so it can't render an unsaved buffer, but nothing in the flow should show a native picker), close the original data file, preview the generated schema (F01), click a field in the rendered HTML, then live-edit the schema and watch the preview refresh (F02). No new fixtures: it reuses showcase/data/person-valid.json, already used by the individual inference demo. An earlier version chained five features (inference, preview/live-update, binding, validation, code generation) reusing the hand-curated person.schema.json rather than the schema the demo itself generated; it was narrowed to this tighter, single-artifact flow once video capture (below) made the compromises in that version — a schema swapped mid-story, an untouched-then-discarded inferred tab — visible instead of hidden by fast-cut screenshot stitching. demo-showcase (command-palette twin) intentionally stays simpler: it previews the existing person.schema.json rather than reproducing the save-dialog-stub and webview-click steps, since its only job is crash-smoke-testing the underlying commands (S08-SR-10), not reproducing the showcase narrative frame for frame. demo-showcase-mouse initially reused the stitched-screenshot approach (helpers/mouse.ts's animated DOM cursor + scripts/make-gifs.mjs) like every other mouse demo, but that read as visibly synthetic once combined into one long recording — the DOM cursor moves in discrete steps (a Playwright-dispatched mouse move never moves the real OS pointer, so it can't be shown in a screenshot at all) and gif-encoder-2's octree quantiser at low quality bands/dithers badly over an extended capture. It was switched to a real screen recording: ffmpeg -f x11grab (helpers/recorder.ts) captures whatever the X server actually composites for the Electron window's content-bounds region, xdotool drives the genuine X11 pointer in lockstep with the UI actions (helpers/realCursor.ts) — cosmetic only, the actual clicks/typing still go through Playwright for the same reliability every other demo depends on — and scripts/make-showcase-gif.mjs converts the recording to docs/public/demo-showcase.gif via ffmpeg's two-pass palette pipeline (palettegen/paletteuse, proper dithering) instead of gif-encoder-2. This is deliberately scoped to this one demo only: the other 16 stay on the screenshot pipeline, which is simpler and has no external-binary dependency beyond the already-required canvas. The refresh-gifs workflow gained ffmpeg/xdotool apt packages and a make-showcase-gif.mjs step. (That last scoping decision was reversed on 2026-09-02 — see "History (encoding)" above: all 17 demos now share the ffmpeg palette pipeline and canvas is gone. The rest of this note still stands.) The preview itself renders json-schema-for-humans' flat template (PreviewWebPanel.ts avoids the default accordion template, which would pull Bootstrap/jQuery from a CDN — a network request this zero-telemetry extension doesn't make, per S05). The flat template has no expand/collapse — "click a field" therefore targets a field far enough down the page that Playwright's actionability scroll-into-view is itself the visible motion, rather than an expand/collapse interaction the real product doesn't have.
• 2026-07 — S08-SR-10's UI-smoke job moved from ci.yml (every PR/push) to refresh-gifs.yml (release time, parallel to the GIF-refresh job it shares fixtures with). Running on every PR meant it contended for shared runner capacity alongside a dozen other concurrent jobs, and made the per-PR CI run noticeably slower for a job that gates nothing. The integration job (S08-SR-08/09) is unaffected and still runs on every PR.
• 2026-07 — Two demos (demo-migrate/demo-migrate-mouse, demo-quickfix/demo-quickfix-mouse) reproducibly timed out opening their seeded fixture via Quick Open (Ctrl+P), on independent runner VMs across multiple runs — a real, repeatable issue with Quick Open's async file-search for those specific fixtures, not the runner-contention flakiness the previous entry blamed it on (that diagnosis was wrong: two isolated VMs hitting the identical failure rules out shared-resource contention as the cause). Fixed by opening the seeded file via VS Code's own CLI launch arguments (runDemo's openFiles parameter) instead of driving Quick Open at all — sidesteps the flaky code path entirely rather than tuning it further. The other 12 demos, whose Quick Open opens are reliable, are unchanged.

Requirement traceability

Status of each requirement in specs/traceability.json. Test coverage is auto-discovered from [ID] tags in test titles and is not listed here.

Requirement	Status	Implementation	Note
S08-NFR-01	manual	.github/workflows/ci.yml	measured against real CI runs (2026-07-13, runs 29279453959 and 29280433507): the "Integration tests" job completes in ~47-52s on ubuntu-latest and ~1m37s-1m50s on windows-latest, including VS Code download/setup, well within the 10-minute budget; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-NFR-02	implemented	src/test/integration/helpers.ts, src/test/integration/suites/singleFolder.remoteSchema.test.ts	remote-schema scenario (jsonschema.cacheSchemaLocally against a "remote" URL) is driven against a loopback-only fixture HTTP server (helpers.ts's startFixtureHttpServer), keeping the run deterministic and offline
S08-SR-01	manual	src/test/integration/runTest.ts, src/test/integration/index.ts, tsconfig.integration.json	runTest.ts launches @vscode/test-electron once per fixture; index.ts is the tdd mocha loader run inside the Extension Host; npm run test:integration; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-02	manual	src/test/integration/fixtures/single-folder, src/test/integration/fixtures/multi-root	single-folder fixture (JSON/YAML/TOML) and a 2-folder multi-root .code-workspace with distinct .vscode/settings.json per folder; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-03	manual	src/test/integration/index.ts, scripts/check-traceability.mjs	suites use suite()/test() (tdd); check-traceability.mjs already walks all of src/test/**/*.ts so [ID] tags in integration suites count with no changes needed; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-04	implemented	src/test/integration/suites/singleFolder.bindingScope.test.ts, src/test/integration/suites/multiRoot.bindingScope.test.ts	WorkspaceFolder/Global x JSON/YAML in the single-folder fixture, Workspace scope and the cross-folder WorkspaceFolder case in multi-root; each asserts both the settings-file path form and that the real validator recognises the binding
S08-SR-05	implemented	src/test/integration/suites/singleFolder.inlineBinding.test.ts, src/test/integration/suites/multiRoot.inlineBinding.test.ts	JSON/YAML/TOML round trip, a TOML file with a pre-existing [table] header, and a cross-folder TOML case exercising absolute-path escaping
S08-SR-06	implemented	src/test/integration/suites/singleFolder.commandSmoke.test.ts	one smoke test per package.json#contributes.commands entry; commands whose real success path needs the network (cacheSchemaLocally/refreshSchemaCache download) or an unstubbed native dialog (preview's showOpenDialog fallback, diffSchema's workspace-file/remote-URL baseline) exercise their nearest safe branch instead — see the smoke suite's file-level comment
S08-SR-07	implemented	src/test/integration/suites/singleFolder.previewPerf.test.ts	measures and logs p95 preview-generation latency over 20 runs against the S03-SR-13 budget; asserts the budget so a regression fails CI, and prints the measured numbers per S08-SR-07's "measured ... and reported"
S08-SR-08	manual	.github/workflows/ci.yml	new "integration" job, continue-on-error (non-blocking) until 2 clean weeks per this requirement; promotion to required not yet recorded; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-09	manual	.github/workflows/ci.yml	integration job matrix runs on ubuntu-latest (via xvfb-run) and windows-latest; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-10	manual	.github/workflows/refresh-gifs.yml, src/test/e2e	e2e-smoke job (parallel to refresh-gifs, both needs: find-pr) runs non-mouse Playwright demos non-blocking at release time — moved off ci.yml per the 2026-07 History note (PR contention caused spurious timeouts); e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-11	manual	.github/workflows/refresh-gifs.yml	e2e-smoke uses --grep-invert mouse; refresh-gifs job uses --grep mouse; each variant runs in exactly one place, both jobs now live in refresh-gifs.yml; e2e/integration process requirement: verified by npm run test:integration in a real VS Code, not unit-taggable
S08-SR-12	manual	.github/workflows/refresh-gifs.yml, scripts/detect-changed-features.mjs	release-workflow requirement: verified by running refresh-gifs.yml, not unit-taggable end to end; the diff/mapping logic it depends on is unit-tested (detectChangedFeatures.test.ts)
S08-SR-13	implemented	scripts/demo-registry.mjs, scripts/detect-changed-features.mjs, scripts/make-gifs.mjs	demo<->spec mapping and its consumers; unit-tested (demoRegistry.test.ts, detectChangedFeatures.test.ts)
S08-SR-14	manual	package.json	policy requirement (test:integration stays unscoped) verified by inspection of package.json/ci.yml, not unit-taggable
S08-SR-15	implemented	scripts/detect-changed-features.mjs, .github/workflows/refresh-gifs.yml	manual demos/all workflow_dispatch override; unit-tested (detectChangedFeatures.test.ts's resolveForceOverride cases)
S08-SR-16	implemented	scripts/make-gifs.mjs, scripts/make-showcase-gif.mjs	one ffmpeg two-pass palette encoder for every demo GIF; the pure ffconcat/frame-ordering helpers are unit-tested (makeGifs.test.ts), the encode itself by running npm run make-gifs
S08-SR-17	implemented	package.json, scripts/bootstrap.mjs, .github/workflows/refresh-gifs.yml	canvas/gif-encoder-2 removed from the manifest; guarded by a makeGifs.test.ts assertion on package.json so the native build cannot come back unnoticed
S08-SR-18	implemented	scripts/make-gifs.mjs	no scale filter in either ffmpeg pass; asserted in makeGifs.test.ts
S08-SR-19	implemented	src/test/e2e/demo-validation.test.ts, src/test/e2e/demo-validation-mouse.test.ts, src/test/e2e/demo-showcase-mouse.test.ts	demo scripts must seed the state their feature needs AND keep the surface the step is about on screen; validation demos seed the binding, demo-showcase keeps the preview beside the editor and asserts it via expectPreviewVisible(); placement/seeding guarded by demoPlacement.test.ts
S08-SR-20	implemented	docs/.vitepress/theme/QuickDemo.vue, docs/guide/commands.md, docs/guide/index.md, docs/guide/lifecycle.md, README.md	every registry demo is embedded in the landing gallery and in its guide section; the README stays showcase-only. Checked both ways (no orphan GIF, no dangling reference) by demoPlacement.test.ts
S08-SR-21	implemented	docs/public	playback duration read from each committed GIF by a dependency-free GIF89a walk in makeGifs.test.ts; showcase is the declared exception

Documented in

Documentation sections tagged spec:S08 (or one of its requirement ids) — 202 of ~1374 expected words (15%). The expectation comes from this spec's evaluated complexity of 34.4 (23 documentable requirements — 21 SR, 2 NFR — weighted by kind and by each definition's length).

• Path-scoped CI/contributing · 202 words

Changes over time

• 909d79e2026-09-09fix(config): create the config file atomically, and rebuild the F08 demo
• ca4bf812026-09-06docs(specs): correct S08 on what a missing demo GIF actually costs
• b2aa8342026-09-06fix(demos): drop the two demos recording proved undemoable
• 41647a52026-09-06feat(demos): add nine missing feature demos and place them in the docs
• 263f6bd2026-09-04refactor(demos): cut the showcase to one story, ending on code generation
• 61489d42026-09-04fix(demos): reach Generate Types through the JSON Schema submenu
• 36c2f912026-09-04fix(demos): let the preview assertion wait for the render it follows
• a3f3e112026-09-04fix(demos): stop aiming the showcase cursor at a hover-only close button
• 636c1e12026-09-03fix(demos): keep the preview on screen through the showcase narrative
• a859f822026-09-02feat(demos): add outline and schema-tests demos, record the coverage audit
• ad93d3a2026-09-02fix(demos): make the validation demo validate, and show every demo somewhere
• 2c0f2b22026-09-02perf(gifs): encode every demo with ffmpeg's palette pipeline, drop canvas
• 8cec0c62026-08-09feat(ci): manual GIF-refresh override, plus a real CI-found fix for the showcase demo (#182)
• 142aa572026-08-03test(e2e): rewrite the README showcase demo into a 13-step tour
• 451ae7b2026-07-26fix(ci): stop reporting failed jobs as green, and earn the check back
• a647c5d2026-07-25feat(specs): per-spec change history, release-scoped demo/GIF selection, concurrent verify gate
• ffb60e32026-07-18feat: add a real-screen-recording showcase demo GIF, embedded in README (#102)
• 93164ea2026-07-16fix(e2e): open quickfix/migrate fixtures via CLI args, not Quick Open
• d39487c2026-07-16chore(ci): move E2E UI smoke off per-PR CI to release-time, parallel to GIFs
• 5dddeaa2026-07-14chore: blog space, site fixes, E2E UI-smoke split, feature demos
• df250472026-07-07fix(S08): green the integration suite — real bugs the first CI run surfaced
• 4b84e532026-07-07docs(specs): add planned specs F18-F20 and S08 (E2E testing)