TreasuryQA Bench
An agent benchmark for office work on scanned records . The agent searches a company Dropbox of 1,435 scanned U.S. Treasury volumes through MCP, reads pages through Reducto’s OCR server, also over MCP, and answers with one exact figure: a value from a table, or a sum, percentage or statistic computed from several.
Leaderboard
Leaderboard
Agent pass@1 95% CI pass³all 3 repeats Clear26 tasks Very blurry20 tasks Answeredcommitted a figure Right thenof those $/task Harness
Claude Opus 5.5 64.8% 55.7–73.3 45.7% 62% 60% 94.3% 68.7% ~$1.56 Claude Code Gemini 4 Argon 60.0% 50.0–70.0 45.7% 55% 50% 72.4% 82.9% ~$2.45 Gemini CLI GPT-6.1 Sol 51.4% 41.0–61.4 38.6% 73% 25% 90.5% 56.8% ~$0.62 Codex Gemini 3.8 Flash 51.0% 40.5–61.4 37.1% 55% 37% 84.3% 60.5% ~$1.74 Gemini CLI Claude Sonnet 5.5 41.9% 31.9–51.9 27.1% 54% 12% 89.5% 46.8% ~$1.19 Claude Code DeepSeek V4.1 Flash 37.1% 28.1–46.2 17.1% 40% 20% 84.8% 43.8% ~$0.25 Pi Grok 4.7 18.1% 11.4–25.2 4.3% 21% 3% 80.5% 22.5% ~$3.55 Grok CLI GPT-5.6 Terra 16.7% 10.0–24.3 8.6% 24% 3% 93.8% 17.8% ~$1.17 Codex
Cost vs. questions answered correctly
10% 20% 30% 40% 50% 60% 70% $0.25 $0.50 $1.00 $2.00 $4.00 agent-side cost per task (log scale) GPT-5.6 Terra: 16.7% pass@1 [10.0–24.3], $1.17 per task GPT-5.6 Terra Grok 4.7: 18.1% pass@1 [11.4–25.2], $3.55 per task Grok 4.7 DeepSeek V4.1 Flash: 37.1% pass@1 [28.1–46.2], $0.25 per task DeepSeek V4.1 Flash Claude Sonnet 5.5: 41.9% pass@1 [31.9–51.9], $1.19 per task Claude Sonnet 5.5 Gemini 3.8 Flash: 51.0% pass@1 [40.5–61.4], $1.74 per task Gemini 3.8 Flash GPT-6.1 Sol: 51.4% pass@1 [41.0–61.4], $0.62 per task GPT-6.1 Sol Gemini 4 Argon: 60.0% pass@1 [50.0–70.0], $2.45 per task Gemini 4 Argon Claude Opus 5.5: 64.8% pass@1 [55.7–73.3], $1.56 per task Claude Opus 5.5 Claude Opus 5.5 Gemini 4 Argon GPT-6.1 Sol Gemini 3.8 Flash Claude Sonnet 5.5 DeepSeek V4.1 Flash Grok 4.7 GPT-5.6 Terra
