Express testing: where the data lives
This is a static export of the evidence behind our A2UI Express test programme (28 and 29 October 2026): raw model replies, saved payloads, scores, the content check and the renderer comparison. It has 2517 files and about 12 MB. Every link is relative, so it works from a local folder and as plain static pages.
Read this first. The samples are small: six briefs (T1 to T6), one to three repeats, one slice per run. The content check is one repeat of 72 cells. Atom-choice F1 is scored against answer key v2 and says which components were used, not what they say. Nothing here ranks models or formats, and nothing here confirms or refutes Google's "up to 80x". Model names (Sonnet, Haiku, Opus, Gemini, Jev, Clef) appear as the subjects that were measured.
Evidence labels
- measured taken directly from a run or a review of the saved output.
- computed derived by arithmetic from measured values (ratios, means, extracted tables).
- sourced taken from a named public page (prices, Google's post).
Files
| File | What it is | How measured | Evidence label | Caveat |
|---|---|---|---|---|
numbers/numbers.json | Report numbers: the eight outcomes (compile rates, output tokens, seconds, atom-choice F1, cost per brief) per model and format. | Generated by a script from the raw results in results/ (this folder). F1 is set-F1 of the atoms chosen against answer key v2. | computed (from measured raw results) | Six briefs, one to three repeats, one slice per run. Atom choice only, not content quality. Gemini 3.5 Flash timing: only the thinking-off re-run is current (see the write-up, section 5, caveat 10). |
numbers/blocks-routing-numbers.json | Routing, cost and token numbers for the curated block agent, the routers and the routed path. | Recomputed from raw results in results/blocks/; prices from the providers' price pages. Includes the 2026-10-30 correction (Jev output tokens are free; earlier Jev rows priced output at the input rate). | computed; prices sourced | Briefs and keys written by us; one writer model (Sonnet 5.5); 3 repeats. The routed path was never run end to end: latency and cost are sums of separately measured means. Uncached and cached cost are different assumptions (cache writes and TTL ignored). |
numbers/blocks-routing-evidence.md | Readable summary of the same blocks and routing evidence (same numbers, same correction). | As above. | computed; prices sourced | Same caveats. Not a ranking of models; nothing here says routing improves accuracy. |
report/writeup.md | The write-up: approach, results tables 3.1 to 3.8, what the results support and do not support, Google claims against our tests. | Tables generated from the raw results; prose written by hand. | measured and computed, labelled per claim in the text | Small sample throughout; the Gemini 1.7x figure in the original film caption is invalid (see corrections). |
corrections/index.html | The corrections to our earlier post (six rows here; the article groups them as three claims and one update), with the figures the data supports (also as corrections.md). | Each row cites its write-up section; figures come from numbers/numbers.json and results/. | computed | One repeat per cell unless stated; small n. |
results/slice-runs/ | Raw results, one file per call (reply text, token counts, seconds, compile result, F1). 166 sets x 6 briefs. Set names read slice, format, model, settings, run. | Calls made with the Anthropic API and Gemini APIs on 2026-10-28 and 2026-10-29: one request per call, no retries on the Gemini runs. Seconds are request time only for sets marked request-only-timing, thinking-off or the Anthropic sets; sets marked first-run-timing-invalid include a rate-limit pause and must not be used for timing. | measured | F1 stored in Direct JSON files misses nested components (re-walk the replies). Haiku 4.5 sets ran at temperature 0 and return nearly the same reply across runs. Opus 5.5 appears here only as a measured writer. |
results/repair/ | Results of one repair attempt on replies that failed to compile. | Original error, rules and component signatures sent back once; result stored beside the original label. | measured | Coarser than the per-line refinement in Google's proposal. Repair time is not in the time figures. |
results/choosers/ | Atom picks per slot by the decision models Jev and Clef (full and flash) on the same saved needs, per slice. | One chooser request per slot; picks scored against key v2. | measured | Only the chooser differs between sets. Times are not comparable (Clef reuses Jev's decomposer output). Oracle slices are a ceiling (built from the key). |
results/jev-layout-pipeline/ | The minimal "80x" build: Jev answers 3 typed layout questions, code builds the screen, a small model writes the text. | 18 screens (6 briefs x 3 runs); Jev request time, writer time and code build time recorded separately. | measured | Three fixed layouts cannot express 5 of the 6 briefs (pipeline F1 0.31). Google's 80x baseline is unstated: neither confirmed nor refuted. |
results/plain-agent-baseline/ | Plain agent with search tools (no catalogue in the prompt), 6 briefs x 3 runs. | Agent loop with catalogue search tools; atoms scored against keys v2 and v3. | measured | Key v3 also accepts filterable_list on T4 (an atom added to close a catalogue gap): never compare a v3 figure with a v2 figure. |
results/blocks/ | Blocks tests: curated 4-atom slice (level 1), with profile (level 2), guardrail round, full-catalog pilot (level 3, 2 calls only), and the routers R and R2 (Sonnet, Jev, non-LLM), with inputs (briefs, answer keys) and scores. | One request per call at temperature 0, 3 repeats; scored by the scripts' rules against keys pinned before the runs. Post-hoc re-scores are labelled as such. | measured | Briefs and keys written by us. One key error was found after the run (house and cat) and is shown as a labelled post-hoc re-score beside the pinned score. Level 3 ran 2 of 24 calls. Atoms named here may be recent additions. |
payloads/ | Saved compiled payloads (the A2UI messages after the compile step), per set and brief. 151 sets. | Written from the raw replies by the saving script after compilation. | computed (from measured replies) | No payloads were saved for the Gemini 3.5 Flash re-runs (request-only, thinking-off) or Gemini 3.8 Flash; use results/ for those. |
content-check/REPORT.md | Content check: 72 cells (12 run sets x 6 briefs, run 1) graded twice, rendered and payload, with causes (model, renderer, test harness), renderer defects R1-R8 and harness defects H1-H8. | First reviewer read each cell's visible text, screenshot at 640 px and payload against the brief; grades good, minor, major, fail. | measured (reviewer judgement) | One repeat, one reviewer per cell. Independence is limited: the first reviewer saw set labels naming the model that wrote each cell and graded cells that include replies from its own model family. No ranking of models or formats is supported. Interactivity untested. |
content-check/first-reviewer-grades.csv | The 72 grades from the report table, one row per cell. | Extracted from content-check/REPORT.md. | computed | As above. |
content-check/second-reviewer/, content-check/second-reviewer-grades.csv | Grades from a second reviewer from a different model family, one raw reply per cell (72), plus a CSV. | Shown only the brief, the visible text and a screenshot of the top 1100 px, no model labels, blind to the first reviewer's grades. One request per cell, no retries. | measured (reviewer judgement) | Agreed with the first reviewer on good-or-minor versus not for 64 of 67 comparable cells (66 within one grade; more generous in 15, stricter in 4; recomputed from the two CSVs). It cannot see payloads, so its cause labels are not used. It also graded cells that include replies from its own model family. |
content-check/screenshots/, content-check/visible-text/, content-check/briefs.json | Inputs the reviewers saw: screenshot per cell (67; five cells have none because the headless browser hung), visible text per cell (72), the six briefs. | Headless browser at 640 px wide, test-harness renderer at origin/main of 27 Oct 2026. | measured | Five T3 cells were judged from text and payload only. |
renderer/index.html | Same saved payloads shown by the 10 Oct 2026 renderer snapshot and by the 27 Oct 2026 renderer, side by side (five cells), with four stills. | Same payload (Sonnet 5.5, Express, oracle slice, run 1) and same harness; only the renderer differs. Self-contained pages, no network requests. | measured (visual comparison) | Five cells; three differ, two look the same. One model, one run. Not a measured rate of improvement. The earlier comparison that mixed in harness fixes is withdrawn and not included. |
MANIFEST.csv | Mapping from the original working file names to the plain names in this folder. | Generated when this folder was built. | - | Original names are kept only as a reference. |
SCRUB-REPORT.md | The pattern search run over every file in this folder before release, with each hit explained. | grep over the whole folder. | - | - |
Naming
Result and payload folders read slice__format__model__settings__runN, for example oracle-slice-3types__express__sonnet-5-5__temp0__run1. Slices: full catalogue, rule slice (family rules), oracle slices (built from the answer key, so a ceiling), tag slice, wrong-domain slice, random slice. Formats: express, json (Direct JSON), elemental. See README.md for method, how to re-score and licence.
Folder sizes
numbers/: 3 files, 25 KBreport/: 1 files, 28 KBcorrections/: 2 files, 7 KBresults/: 1579 files, 4.7 MBpayloads/: 695 files, 2.0 MBcontent-check/: 218 files, 4.1 MBrenderer/: 15 files, 684 KB