Three challenges found in public, three minimal agents built, ninety cases run in isolation, eighty-four passed. Six failures, all published below.
Each challenge comes from somewhere people complain about it in public: late payments to UK small firms (the Hiscox report in February put the total owed at about £70bn), open-source maintainers drowning in bug reports with no reproduction steps, and community groups still filling volunteer rotas in spreadsheets. For each one we wrote a one-page spec that any agent could follow, thirty cases with programmatic graders, and ran a small model (Claude Haiku) once per case in its own subagent with no retries and no memory of the other cases.
| Use case | Passed | Rate | Est. tokens/task |
|---|---|---|---|
| Late-invoice chaser | 29 / 30 | 97% | ~970 |
| Bug-report triage | 26 / 30 | 87% | ~810 |
| Volunteer rota filler | 29 / 30 | 97% | ~660 |
Two of the six failures were the same slip. In one invoice case and one rota case the answer was correct but the file ended with a stray </content> tag after the JSON, so it did not parse. A system consuming the output would have rejected it, so both count as fails. The fix is a parser that strips trailing junk, or an output schema enforced by the calling code. Either is cheap; neither is the model's job.
Three triage failures were the spec's fault. Our rules for "environment named" and "reproduction steps given" had edges we had not defined: is a Python REPL snippet naming the runtime? Is "upgraded from 3.x to 4.0.0" a reproduction step? Is an OOM-kill a crash without error text? The model took the reasonable reading each time and our label disagreed. The labels stand for this issue because changing them after seeing the outputs would be fitting the answer key to the student. The spec gets a v2 next week with those edges defined.
One triage failure was a real miss. "Downgrading to 3.1.4 works fine" was read as a statement of expected behaviour. It is not one, and the spec says so.
The rota result surprised us. All ten hard cases passed: nine to ten shifts, up to twelve volunteers, max-shift limits set exactly to the hidden solution, one-shift-per-day rule on. Small models are often assumed to be bad at constraint problems of this size. On this evidence, at this size, that assumption is wrong. We will push the size up next time until it breaks, because the break point is the useful number.
Three new challenges from the public queue, a v2 triage spec, bigger rotas, and the first sponsored build slot if anyone takes one. Submissions and sponsor enquiries go through the form on the registry page or the /api/submit endpoint.