A small methods update on the Jev/A2UI results

After publishing Testing Jev, which found that the hand-written shortlist, rather than Jev alone, accounted for most of the reported accuracy lift, I went back through the retrieval and selector layers separately.

First, retrieval. On the article's existing needs, keyword retrieval with a top-12 pool surfaced an accepted atom 83% of the time. Vertex AI embeddings raised that to 91% at top 30 and 96% at top 50.

The same direction appears end-to-end on the article's six briefs: embedding retrieval improves open-pool F1 for Haiku, Sonnet, Gemini and Gemma 3 4B. This is the practical route from a hand-written shortlist to an automatic one.

The selector question lands differently: on the data available, F1 comes out about the same whether the model picks the atom itself or Jev picks it. In the Haiku + embeddings slice, self-selection scored F1 0.432 against Jev's 0.428 — essentially matched, not a rounding trick. They make different mistakes on the way to that number, but neither selector is ahead.

The hand-written shortlist tells the same story with one small, real exception. I completed the missing shortlist + self-pick comparison, then traced an apparent Haiku self-pick lead before treating it as a result — because it was too good to be the real finding, and it wasn't. It was not a chooser effect: two required slots were absent from the Jev run's independently sampled decomposition before Jev was ever asked to choose an atom for them, so the "lead" was slots Jev never got a shot at, not slots it lost. Across the 15 slots both paths actually reached, the selectors agreed on correctness in 14. The one reproducible atom-choice difference — the same one, both in this run and in an earlier isolated-choice check — favoured Jev, by a single slot. That's the whole gap: matched almost everywhere, with one small, repeatable exception in Jev's favour, not a systematic advantage either way.

That surfaced a more basic evaluation rule: a selector cannot choose a component for a slot the upstream decomposer never surfaced.

Future direct chooser comparisons in this project now use a paired harness: decompose once, freeze the slots and candidate lists, then pass the identical decision to every chooser. This is enforced by construction rather than checked afterwards. The experiment inventory records the stage boundary, candidate source, decomposition status, scoring route and artefacts for every run.

A third chooser: Cloudflare's Clef

Cloudflare released two open decision models, Clef and Clef-flash, on 1 October. They take the same kind of request as Jev (a state, a typed question, a set of options) and return a probability for each option, so I swapped one file in the existing harness and left the rest alone.

On the six briefs I used the paired harness this post describes. Haiku decomposes each brief once and the slots are matched once. Jev, Clef-flash and Clef then each choose from the identical slots and the identical hand-written candidates, and the selection is scored on its own, with no content generation. Three repeats, 18 cells per chooser. Clef-flash scored the same as Jev in every cell (F1 0.93 for both), although its picks were identical in only 9 of the 18: it chose data_table_sortable, for example, where Jev chose table. Clef picked a different atom from Jev in 12 cells. Every atom any of the three picked is one the briefs accept (precision 1.00), and only one difference changed a score: on the KPI-tile slot of the dashboard brief Clef chose stat_card where Jev chose metric_row, in all three repeats. Both are accepted for that slot, but metric_row makes the brief's trend requirement apply and the scorer docks it, which accounts for the whole gap (0.96 against 0.93). As Testing Jev found, a hand-written shortlist is a weak test of the chooser, so this says Clef is not worse than Jev there and nothing more.

My first attempt did not use the harness. I ran Clef through the earlier end-to-end setup, with its own decomposition, and it scored 0.98 against 0.91 for the Jev run. With the decomposition held fixed the same two choosers are at 0.96 and 0.93, which fits the decomposition being the cause of the first gap, though I did not run an ablation to prove it.

With no shortlist, 150 single-request prompts from the Worker's routing eval each asked for one atom out of the 540. Clef-flash picked an accepted atom 86.7% of the time and Clef 83.3%. That is a different yardstick from the open-pool F1 above, and Clef had to run as three groups plus a final round because it accepts at most 255 options, while the models I compared against (the best, Llama 3.3 70B on Workers AI, scored 85.3%) saw all 540 descriptions in one prompt. What I find most useful is the confidence. When Clef-flash was at least 80% sure (62% of requests) it was right 94.6% of the time. That gives the optional selector below something to do: take the chooser's answer when it is sure, and ask a language model only when it is not.

Then I ran the paired, open-pool test. Haiku decomposed each brief once per repeat, each pool was built once, and Jev, Clef-flash and Clef chose from the identical pool: three repeats over the six briefs, on a keyword pool and on an embeddings pool. All three scored about 0.4 on both, and every difference from Jev was within noise (no larger than 0.02, every interval spanning zero, wins and losses close to even). The means do hide task-level differences that look consistent: on one brief both Clef models trail Jev by about 0.20 in both pools, and on another they lead by about 0.10 to 0.19, and the two roughly cancel. So Clef matches Jev here and does not beat it, and the open-pool number is limited by the pool and the decomposition, not by which of the three chooses. Six briefs and three repeats can rule out a large gap, not a small one. The full tables, the other models and the cost are in the project notes.

The architecture I am taking forward is:

brief → decomposition → embedding retrieval → bounded candidate pool → optional selector → A2UI payload

Embeddings solve more of the open-pool problem than routing alone. The remaining work is decomposition reliability, catalogue descriptions for confusable atoms, and held-out generalisation — not another round of headline claims.