A grounding benchmark for spreadsheet agents. 96 locked tasks, three failure modes scored mechanically, no LLM judge anywhere in the scoring. Every model reply on this page is a real recorded output from a real run.
Four 3.35B models Cohere Labs ships, given the same sheet and the same prompt. Pick a task and watch them disagree.
p1_control · curator-1-002: the
sheet plainly contains March sales of 13500, fire and
water return it, and earth reports the data missing.
Then try any p2_ambiguity task, where global picks
one of two identically-headed columns without saying that it chose.Token counts and throughput are measured from this benchmark's own 96-task runs, not estimated. Put in your volume and your rate.
The scorer is mechanical, and you can drive it. Pick a task, write what a model might reply, and see which metric fires. Invent a number on an unanswerable task, or add an edit to a cell outside the write-set.
node validate.mjs to check it yourself.