TaxClaw benchmark
This is the benchmark for the TaxClaw Extraction Standard, core version 1.0, schema version 1.0. Every signed TaxClaw receipt names that standard and links to this page (published receipt metadata).
- A TaxClaw receipt proves origin and integrity: TaxClaw Cloud produced that exact output from that exact input document under the stated standard, and the output has not been altered since. It never proves the extracted values are accurate.
- Every figure on this page was measured on synthetic (invented) documents. None of it is real-world accuracy or tax-preparation accuracy, and none of it is a promise about your documents.
- Check every extracted value against the source document.
Open-source TaxClaw 1.0.0
These results come from the “Measured accuracy (synthetic benchmark)” section of the TaxClaw 1.0.0 CHANGELOG. TaxClaw is assisted extraction: a person reviews the output.
Release gate, 2026-10-21 (application b2c4f4c)
13 supported form types, each as a clean PDF and as an image-only degraded scan: 26 documents, 562 graded fields (281 clean, 281 degraded). Best-effort Form 1040 is not part of this gate. Evidence: RESULTS.md (on the benchmark branch, commit 3dcc5a1) and PR #10 comment 6310221862.
| Path | Model | Clean | Degraded scan | Total |
|---|---|---|---|---|
| Subscription CLI adapter (the gate result) | claude-haiku-5-5 | 281/281 fields, 13/13 exact | 281/281 fields, 13/13 exact | 562/562 fields, 26/26 exact |
| Production openai_compat backend, run 1 | anthropic/claude-haiku-5.5 (OpenRouter) | 280/281 fields, 12/13 exact | 281/281 fields, 13/13 exact | 561/562 fields, 25/26 exact |
| Production openai_compat backend, run 2 | anthropic/claude-haiku-5.5 (OpenRouter) | 280/281 fields, 12/13 exact | 281/281 fields, 13/13 exact | 561/562 fields, 25/26 exact |
- All three runs classified 26/26 documents correctly and had zero runtime failures.
- The subscription CLI adapter is not one of TaxClaw’s production API backends, so its result says nothing about the production path.
- The production runs used TaxClaw’s own
extract_document_with_metacode on the same 26 fixtures. Each run missed one field in one clean document. Run 1: 1099-DAaggregate_flagreturned1where the form is blank. Run 2: K-1other_deductions.13Areturned null instead of 125.00. Each run cost about $0.02 on OpenRouter. - The production path therefore scores 561/562 fields and 25/26 exact documents per run and does not meet the strict all-exact gate.
Changed-form run with local models, 2026-10-20 (application 87efee5)
Eight invented documents per model (1099-R, 1099-G, 1099-K and K-1, each clean and as a degraded scan), 224 graded fields. Local-model numbers come only from this run; the 2026-10-21 gate did not include local models. Evidence: RESULTS.md.
| Model | Fields | Exact documents | Seconds per page |
|---|---|---|---|
| claude-haiku-5-5 (subscription CLI wrapper) | 224/224 | 8/8 | 5.21 |
| gemma3:12b (local) | 176/224 (78.6%) | 0/8 | 65.95 |
| gemma3:4b (local) | 158/224 (70.5%) | 0/8 | 18.88 |
- No local model produced an exact document: 0/8 for both
gemma3:12bandgemma3:4b. - The claude-haiku-5-5 row ran through a subscription CLI wrapper, not the production API path or its privacy controls.
- Other local models in the same run:
glm-ocrclassified both K-1 documents as 1099-DIV (classification 6/8);qwen3.5:4bhad 5 failed extractions (empty or truncated JSON);gemma4:latesthad 1 extraction error (no response after 180.11 s). - An earlier 12-form strict run on a different corpus scored lower: claude-haiku-5-5 190/233 (81.55%) frozen and 193/233 (82.83%) exploratory, gemma3:12b 177/233 (75.97%), gemma3:4b 155/233 (66.52%). It reports no model passing its strict gate and makes no release recommendation. Evidence: RESULTS.md.
Not covered
Clairic was not benchmarked. The Anthropic API backend has not been run against the benchmark. Of the openai_compat presets, only OpenRouter has full benchmark runs (the two above); the others are tested with mocked responses only.
Hosted service (TaxClaw Cloud, mcp.taxclaw.net)
TaxClaw Cloud reads each page with a primary model (OpenAI gpt-6-luna) and a cross-check model from a second provider (Anthropic claude-haiku-5-5), and sends the page to an escalation model (OpenAI gpt-6-sol) when the two reads disagree or a read fails. A judge step between cross-check and escalation is described but not switched on yet. See the TaxClaw Cloud Privacy Policy, section 4.1.
This hosted routing has not been benchmarked as a pipeline yet, and this page has no figures for it. The open-source results above were measured on invented documents under test conditions that differ from the hosted service’s production routing. They are not a measurement of accuracy on real documents or of the hosted service as deployed (TaxClaw Cloud Terms, section 5.4). A hosted benchmark will be published on this page, with a version number, when it has been run.
Known limitations
- Local models are for assisted extraction, not unattended use. Small local models such as gemma3:4b can misread checkboxes and miss values such as wash-sale adjustments.
- TXF can’t carry 1099-DA boxes G–L. TXF v042 has no codes for them, so those sales are skipped with a warning and appear only in the Form 8949 CSV.
- Confidence values and
needs_reviewflags are aids for review, not promises. A value can be wrong without being flagged. Check every value, flagged or not.
Last updated 2026-10-26. Questions about this page: lando@outbranch.net. Plain-text version: /benchmark.md.