105 real cases, measured against the model alone.

The short answer

• Things wrongly tagged Do not use: 0 by Tagout, against 5 by Nemotron alone and 5 by Nemotron reading the same web results without rules.
• Tagout caught 46 of 46 recalled units with a red tag; Nemotron alone named 5, and 29 with search.
• Recalled things cleared green, the error that matters most: 0 by Tagout, 36 by Nemotron alone and 12 with search.
• A check cost 2.6¢ per item on Nemotron 3 Ultra and took a median 23 s.

Does the agent earn its place?

The same 105 cases, five ways. The model alone answers from memory. With Tavily it reads the official recall pages a web search finds, but decides on its own. The fixed plan runs set searches instead of the agent, with Nemotron 3 Nano reading the notices and the same rules deciding: that is now Tagout's lite tier. The last one uses no model at all, which is what the demo used to fall back to.

Every case, every arm

One mark per answer, in the tag's colour. A crossed mark is a wrong answer. Hover or tap a row for the line that was typed.

What Tagout got wrong

1 of 105 answers. Each one, and why.

• k04 Shark Navigator upright vacuum, model NV352 Check the label

  A 2015 CPSC recall of four Shark Rotator vacuums came back, and the notice reader returned none of their model numbers, so the matcher couldn't rule this Navigator out and asked for the label. An orange that wasn't needed; nothing unsafe was said.

The first run, and what it changed

The first full run got 98 of 105 right: 2 units wrongly tagged Do not use and 2 recalled units cleared. Each miss was traced to its cause and fixed in code, with a unit test, before the run above. The numbers above are therefore measured on cases that shaped the fixes; the first run is the fairer estimate for things Tagout hasn't seen.

• p11 Cuisinart grill brush CCB-5014 (and p22, a Rowenta vacuum): the notice reader returned no model numbers for a CPSC notice that prints them, so the tag stayed orange. Fixed: a code of yours that the notice's own text prints, as a whole word and outside any "not included" sentence, now counts as listed.

• d06 GenTeal Tears lot 4V15 was cleared. FDA prints the lots in groups ("9T20, 9T50 … Exp Date 4/30/2026; 4V15, Exp Date 8/31/2026") and the lot list stopped at the first group. Fixed by the same text check; this was the most serious miss of the first run.

• f01 Lundberg rice lot 260201MA stayed orange. The agent read FDA's web page for the recall, and that page (mostly navigation) replaced the record's own text, lot list included. Fixed: a page read for an agency record is added to its text, never swapped in.

• n04 A Love To Dream sleep machine with date code 7W16TN was tagged Do not use. The recall covers model LTD-SM23 only with date code 7W15TN, but the reader filed no date codes. Fixed: when a notice limits a recall to lots or date codes and yours isn't printed in it, a model match is orange, not red.

• n12 A Thermos SK3000 made in 2024 was tagged Do not use; only jars made before July 2023 are recalled. The reader answered the window as "July 2023" with no start date, and only full ISO windows were used. Fixed: one-sided windows and month dates are read.

• c08 Clear Eyes Redness Relief drops with no lot were cleared. openFDA's brand field is empty on the Sept 2026 recall, so the brand search missed it, and the agent dropped the record a word search found. Fixed: drug brands are also searched in the description, and the brand's FDA recalls of that product are kept when no lot was given.

• n21 Found while re-running after those fixes: Gas-X lot YH9Z was tagged red against lots YH9X and YH9Y, because a trailing X counted as a family wildcard. Fixed: in lot lists an X is a letter.

• k07 Ground truth corrected: we had Claritin as never recalled, but a Jan 2026 FDA recall covers all lots of Claritin 20-count distributed by Gold Star Distribution. Tagout's orange was right; the case now accepts orange or green.

Which Nemotron for which job

Tagout has three model jobs: reading the list, running the search agent, and reading each notice's codes. The track asks for Ultra where the reasoning is hard and smaller models for the everyday calls, so every size ran every case.

Measured, then routed. Nemotron 3 Ultra stays on all three jobs while the day's demo budget lasts: as the notice reader, Super wrongly tagged one unit red and Nano two; as the list reader, Super filed eye drops and food as products (the wrong database) and Nano dropped a car's year. On 18 cases, a Super search agent was right on 17 at about half the cost, and Nano on 15; that is too small a sample to move the agent. What did move: when the Ultra budget runs out, Tagout used to fall back to no model at all (87 of 105 right, 10 recalled units cleared). It now falls back to the fixed search plan with Nemotron 3 Nano reading the notices: 98 of 105 right, none cleared, at 0.29 cents an item, a ninth of the full check.

How it was measured, and what it can't show

• Cases come from CPSC recalls of 2025–26 (SaferProducts.gov), openFDA enforcement reports and NHTSA's recall API. eval/verify.mjs checks each one against the agency record without any Tagout code: a recalled code must be printed in its notice; a near miss must not be printed in that notice or in any CPSC or FDA record of the same brand.
• Each arm ran each case once, on 15 Oct 2026. Model answers vary between runs; this is one sample, not an average. Every HTTP response was cached, so all arms read the same agency data and the whole run can be replayed for free.
• Two changes made after this run are not in its numbers: an agency's page listing many recalls no longer stands in for one notice, and an FDA company announcement with no enforcement report behind it is marked as ahead of the database.
• We chose the cases. Recalled units come from recent notices that print a code, which is the case Tagout is built for; a product whose notice prints no code can only ever be orange.
• Cost is tokens times Token Factory's list prices ($1/$3 per million for Ultra, $0.30/$0.90 for Super, $0.06/$0.24 for Nano). Agency APIs and Tavily cost nothing here.

Reproduce it: node eval/verify.mjs, then npx tsx --conditions=react-server eval/run.ts tagout for each arm, then node eval/score.mjs. Code and cases: github.com/RohanGlitched/tagout/tree/main/eval. Check your own things.