Cellprose benchmark protocol

Generated from docs/PROTOCOL.md. Edit it there.

Draft, Oct 9, 2026; baselines finalised Oct 11, 2026 (LlamaSheets dropped). Fixed before the final run; any change after that run starts is logged in the write-up. Items marked To build are not implemented yet and block that run; items marked Built have code and tests but have not been run against a live API.

This document says how the published benchmark is run, scored and reported, so the result does not depend on choices made after seeing the numbers. Exploratory runs (E01–E18) do not have to follow it; the one publishable run does.

1. Question

For each model, does Cellprose v0.2 let the model answer questions about a workbook correctly for less money than the two baselines a user would otherwise reach for?

Decision (Oct 9, 2026): the primary baselines are the real tools, not our own CSV rendering.

Role	Baseline	What it is	Runner
Primary	markitdown	Microsoft MarkItDown 0.1.7's own output for the .xlsx, sent in the prompt (one-shot)	api_run.py
Primary	agent_xlsx	The .xlsx on disk and an agent with bash and read_file tools (python3 and openpyxl available), exploring until it answers	api_agent.py
Reference	csv	Sheet-per-section CSV, one-shot. Kept so the earlier experiments (E01–E18) stay comparable	api_run.py
Secondary	markdown, toon	Our imitation of MarkItDown's layout, and TOON	api_run.py

• Compared encoding: cellprose_lean_gfm (the v0.2 format, src/cellprose/writer.py), one-shot, against each baseline above, per model.
• Also reported, not tested: agent_cellprose (the v0.2 text as a file read through the same tools), the other encodings in encoders.ENCODINGS.
• Control: none (questions without the workbook). It shows how much can be guessed; it is not compared.
• Not included: LlamaSheets. The service was retired in August 2026, so there is nothing to run and no placeholder for it. The baselines above are the final set.
• Caveat on the agent baseline: a one-shot call and an agent loop spend tokens differently (the loop re-reads a growing context every turn). The comparison is on cost per correct answer, which is what a user pays, but a win over agent_xlsx says nothing about which of the two reads a workbook better at equal cost.

2. Metric

The headline metric is cost per correct answer, per model:

cost per correct = mean cost per call / mean accuracy
cost per call    = (all input tokens × input price + all output tokens × output price) / 1e6

• Output tokens include reasoning tokens.
• Every input token is priced as uncached input, whatever the provider reports. Prices come from runner/models.py and are recorded with the run.
• Accuracy is the mean question score (section 6).
• Also reported per model and encoding: accuracy, input tokens, output tokens, and cost per call.

3. Test set

Set	Source	Role	Status
Synthetic	14 workbooks from gen_workbooks.py and gen_hard.py, 4 of them "hard"	Development	Exists
SpreadsheetBench	Cell-level tasks from verified_400, restated as value questions by external/spreadsheetbench.py (29 workbooks, 120 questions, 342–18,613 cells)	Held-out	Exists; not in git (CC BY-SA 4.0)
Real-world	17 public workbooks (4 Damodaran valuation models, 13 government data files), 67 questions, named rw_<name>; built by benchmarks/external/realworld.py (external/realworld/)	Held-out	Built; questions read by one reviewer, see its review.md

Rules:

• Development data may be used to tune the format, prompt and runner.
• Held-out data is run only in the final run. If a format, prompt or scoring change follows a look at held-out results, those results are exploratory and a new held-out set is needed for the claim.
• Built: the split is benchmarks/split.json, committed before the final run. It lists the development workbooks; anything not listed (every sb_* and rw_* workbook, and any workbook added later) is held out.
• Built: api_run.py and api_agent.py refuse a held-out workbook unless --final is passed, before any call is planned or priced. --final also refuses a working tree with uncommitted changes (--allow-dirty overrides it, and the flag is recorded). A default run uses the development workbooks only.
• Built: --subset N draws N held-out workbooks with a seeded random draw (--seed, default the seed in split.json, fixed before any held-out result existed). The seed and size are recorded with the run.
• Workbooks, questions and ground truth are frozen at a git commit. Changing any of them makes earlier results incomparable and is stated in the write-up.

4. Models and settings

Each model is a pinned alias in runner/models.py with an exact model ID, list price, max_tokens, and an explicit thinking or effort setting. Temperature is left at the provider default. claude-haiku-5-5 and claude-sonnet-5-5 have no dated snapshot id (the API returns the alias), so a run is tied to its date and runs from different days are not compared. Results are reported per model and never pooled across models.

Proposed model set, to be confirmed by the owner with the cost estimate (section 8). A smaller plan is under discussion (not decided): four models, Haiku 5.5, Sonnet 5.5, GPT-6 Luna and GPT-6.1 Sol, two repeats and a seeded held-out subset, all of which section 9 allows.

Vendor	Cheap	Mid	Top
Anthropic	haiku55	sonnet55	opus55
OpenAI	gpt6luna	gpt61sol	gpt6astra

• fable51 is registered but parked.
• haiku45 is reported separately if run; it is never compared with Haiku 5.5.
• Built: the OpenAI-compatible provider (runner/providers.py) runs open-weight models on any Chat Completions endpoint (CELLPROSE_COMPAT_BASE_URL). No open-weight model is chosen, so none is registered: the owner picks the model and host, and the host's list price is passed as --openweight alias=model_id,price_in,price_out. It is one-shot only (no agent mode), and caching is whatever the host does.
• Built: prices were checked against the vendors' own pricing pages on 2026-10-11 (Anthropic and OpenAI; all eight registered models match what runner/models.py held). The check found a long-prompt rate that the registry lacked: Haiku 5.5 charges $0.50 / $2.50 per million for a request whose prompt is over 100,000 tokens, and the GPT-6 models charge more above 272,000. The registry now carries these tiers, and cost is computed per request (per turn in agent mode). Re-check the pages on the day of the final run and record the date with it.

5. Procedure

1. Prompt. Every call uses runner/prompts.py unchanged: the same system prompt and rules for every encoding. Only the workbook block and its format label differ. All questions for a workbook go in one call, and the answer is one JSON object. Agent mode uses AGENT_SYSTEM and agent_prompt in the same file: same questions and answer rules, with the workbook named as a file.
2. One invocation. Every model, encoding, workbook and repeat runs in a single api_run.py invocation with one run id. Results from different invocations are never compared, even for the same model.
3. Repeats. Three repeats (--reps 3), repeat-major, so a provider change mid-run hits every encoding alike. Two repeats is the lower bound under discussion; it is a stated scope cut, not a default.
4. Caching.
   • Anthropic calls send no cache_control, and any cache read or write is an error (in agent mode too, on every turn).
   • OpenAI caches automatically and can't be turned off. Cached tokens are recorded and priced as uncached.
5. Recorded per call: requested and returned model ID, settings, the prices used, raw usage, cost, run id, git commit, dirty-tree flag, and the SHA-256 of the prompt sent.
   • Built: each run writes <out>/runs/<run id>.json with the run block, how it was invoked, and the SHA-256 of every question file, encoded file and (agent mode) .xlsx it can send, plus one digest over all of them and the SHA-256 of split.json. The manifest is small and is committed with the run id, including for SpreadsheetBench-derived inputs, which are not.
   • Tokenizer counts, built: next to billed usage, each Anthropic call stores the count from Anthropic's count_tokens endpoint for the system prompt and prompt, and each OpenAI call the o200k_base count from tiktoken. analyze.py prints them per model and encoding, with the ratio to CSV, so a saving can be checked against each vendor's own tokenizer. Open-weight endpoints give no count. Agent calls have none, because the context changes every turn.
   • Agent mode records the tokens of every turn and the commands run (run.provider_usage.turns, tool_trace), at most --max-turns (default 40) turns and --max-call-usd (default $1) of list-price spend per call.
6. Failures. A call that errors is retried by the SDK (3 retries).
   • A call that still fails, is truncated at max_tokens, or is refused scores 0 on every question.
   • Its tokens, where reported, count toward cost.
   • Failures are listed per model and encoding in the write-up.
   • Built: the runner writes a failure record (failed: true, with the kind, e.g. truncated, refused, cache_leak, api_error, and in agent mode turn_limit, budget_exceeded) and score.py scores every question 0. A bug in the runner itself is not recorded as a model failure. A resumed run treats a failure record as done; it is not retried.
   • Agent mode adds two failures of its own: using every turn without a final answer, and reaching the per-call spend cap. Both score 0 and keep the tokens spent.

6. Scoring

Scoring is score.py, unchanged from the exploratory runs:

• The last JSON object in the reply whose keys start with q is the answer. Missing or unparsable answers score 0.
• Value questions: numbers match within 0.5% relative or 0.01 absolute, whichever is larger. Strings match case-insensitively after trimming.
• Address questions: the set of cells must match exactly.
• List questions (trace, impact, hardcode): F1 between the expanded cell sets. Both empty scores 1.

7. Statistics

• Unit: one call (model × encoding × workbook × repeat). Calls for the same workbook and repeat are paired across encodings.
• Interval: a 95% interval for the ratio v0.2 cost per correct / baseline cost per correct, per model and per baseline, from a paired bootstrap that resamples workbooks (10,000 resamples, seed 20261009, percentile interval; repeats are averaged within a workbook first). Accuracy differences get the same treatment. Built: stats.py, printed by analyze.py and saved as verdicts.json.
• A model counts as a win when the whole interval for the ratio is below 1 and v0.2 accuracy is no more than 1 point below CSV.
• A loss is an interval entirely above 1. Anything else is no difference.
• Every model is reported as a win, loss or no difference against each baseline (the primary ones first). No model is left out, and no claim is made across models.
• A ratio is infinite when a side answers nothing correctly; such a side counts as a loss for that side.
• Synthetic (development) and held-out sets are reported separately by analyze.py, using split.json. The headline uses held-out.

8. Cost and spending

• api_run.py prints the expected cost and the worst case (every call uses max_tokens) before any call.
• The owner approves any run expected to exceed $5.
• --vendor-cap-usd stops new calls once a vendor's spend reaches the cap. A run stopped by a cap is incomplete and is not published. Resuming creates a new run id, so the whole run is redone.
• Rough size of the final run (inferred from E18's estimate, not measured):
  • The held-out SpreadsheetBench set alone, at 2 encodings × 7 models × 1 repeat, came to about $89 expected.
  • With 4 compared encodings and 3 repeats, the cost grows about 6× (roughly $500–550). The none control adds little because it sends no workbook.
  • Section 9 lists ways to cut it.
  • That figure predates the baseline decision of section 1. api_run.py --estimate-only and api_agent.py --estimate-only print the current estimate for any plan; the agent one is rougher (section 5, item 4 of the to-build list) and its worst case is the per-call cap times the number of calls.

9. Scope cuts allowed before the run

The run may be made smaller, but only before it starts and only by these levers, each stated in the write-up:

• fewer models (keep at least one cheap and one top model per vendor);
• secondary encodings dropped (never markitdown, agent_xlsx or cellprose_lean_gfm; csv is the reference and is dropped last);
• none run once instead of per repeat;
• a fixed random subset of held-out workbooks, drawn with a recorded seed.

10. Reporting

The write-up is benchmarks/RESULTS.md plus one experiments/ file. It contains:

• method (this document and the commit it was frozen at), models with exact IDs and prices, run id, and dates;
• per model: the table of section 2 for every encoding, the cost-per-correct ratio with its interval, and the verdict;
• the headline chart, one bar per model with its interval;
• failures, scope cuts, and every deviation from this protocol;
• limitations next to the headline number: test-set size, the share of synthetic data, the date the prices were checked, the one-shot against agent-loop caveat of section 1, the open-weight model and host chosen, and that savings may differ by tokenizer (section 5 lists the counts). The wording lives in benchmarks/limitations.py; report.py and analyze.py print it under their tables, and a test checks that the README and DESIGN.md carry the short form.

Raw responses for the synthetic set are committed. SpreadsheetBench-derived files stay out of git because of the share-alike licence; their run ids and checksums are committed instead.

11. Not covered

• Batch APIs. Batch discounts are not used: the runner makes direct calls only. If a discount is wanted, the price used for cost per correct stays the list price, so the write-up would state both.
• Open-weight models in agent mode. The agent loop supports Anthropic and OpenAI only.
• The compact model mode (pipe tables sent as comma grids) was deferred on Oct 9, 2026; v0.2 is benchmarked as stored.

To build before the final run

1. Real-world held-out workbooks with hand-checked questions (section 3). Built (backlog B9). A second reviewer should still read external/realworld/review.md before the final run.
2. Verified GPT prices (section 4). Done 2026-10-11; re-check on the day of the final run.
3. Owner sign-off on the model set, the open-weight model and host, and the cost estimate (sections 4 and 8).
4. A small live check of the agent runner on both vendors, priced first. The agent loop and the OpenAI Responses tool loop are built against fake clients and, in tests/test_agent_wire.py, the real SDKs with the HTTP layer mocked (request bodies, streaming and response parsing, history resend); no real call has been made, so the first one may need adjustments, and the per-turn token averages in the agent cost estimate come from the earlier CLI runs, not from this loop.
5. A decision on batch pricing (section 11).