Nanonets' own benchmark says the best AI agent finishes just 38% of multi-step enterprise tasks
Nanonets, a private document-AI company whose platform automates paperwork for 10,000-plus teams, published an open benchmark grading AI systems on executing finance processes under rules buried in conversation; the best result was 38% of tasks fully right. Against $2.7 trillion of forecast 2026 AI spending, it is a rare inside-industry measurement of what enterprise agents actually deliver.
Vincent Jiang · 3 min read
A number sitting in public since July
The best AI system Nanonets could find finishes only 38% of multi-step enterprise process tasks fully right. The number sits on the research page of Nanonets, a private San Francisco document-AI company, under TRACE, Task-Relevant Applied Constraint Execution, an openly published benchmark 1. It is a ceiling on the work behind $2.7 trillion of forecast AI spending in 2026 3. The write-up's title states the finding: models find the rules, then fail to apply them 2.
The dataset and its scoring code were last updated July 22, 2026 67. News and web searches on October 14, 2026, found no outlet reporting the result.
The money assumes the opposite
Gartner says software vendors of every type are "rapidly embedding agentic AI" in their existing products 3. Nanonets says more than 10,000 teams already run invoice matching, claims and month-end close on its platform 4. If the best system passes 38% of process tasks, most budgeted automation work still comes back wrong, or comes back to a human.
If 38% holds, two groups lose: enterprises deploying agents on vendor claims, and software revenue models priced for near-total back-office automation. The winner is whoever first posts a verified execution score far above it.
The grader also sells the agents
Nanonets is not a neutral referee. It grades the field, running TRACE and a public leaderboard that ranks 29 intelligent document processing models, with its own OCR-3 at number one 15. It also sells into that field: a blog marketing "AI-native" platforms against legacy vendors ABBYY and Kofax 4, and a banner for Trail, "the company brain for AI agents" 1.
What the test asks
TRACE is deliberately harsh. Each sample is a Record-to-Report conversation, the back-and-forth of a month-end accounting close, about 100,000 tokens of text long, with company rules dropped mid-sentence. Five samples carry 47 questions graded on 388 criteria, scored all-or-nothing: a task passes only if the answer is right and every rule that applies is honored. A second AI model does the grading, with gpt-5.5 named in the sample setup 6.
Reading in the 80s, execution at 38
On Nanonets' own document leaderboard the same models read pages far more reliably: Nanonets OCR-3 at 85.9 overall, OpenAI's$1.18T — OpenAI, private, latest valuation $1.18T GPT-5.4 at 83.5, Google's Gemini-3-Pro at 82.8 5. For a buyer, the two scores together mean a model can read a page right roughly 86 times in 100 and still fail most multi-step processes it runs under a company's rules 1.
Reading documents is nearly solved; the best execution score is 38
Data
| Value | |
|---|---|
| Nanonets OCR-3 (reading) | 85.9% |
| GPT-5.4 (reading) | 83.5% |
| Gemini-3-Pro (reading) | 82.8% |
| Claude Sonnet 4.6 (reading) | 80.7% |
| Best system tested (execution, TRACE) | 38% |
Small benchmark, interested party
The caveats are real: five samples, 47 questions, one context size, self-published by a vendor with agents to sell, machine-graded 6. But the dataset is open under CC-BY-4.0 with the evaluation code on GitHub, so anyone can rerun the field instead of trusting the number 67. Almost nobody has: the dataset card shows 44 downloads in the past month 6, the repository shows 39 stars and no forks 7, against 288,000 downloads for Nanonets' own OCR model 8.
Almost nobody has rerun the benchmark: 44 downloads against 288,000 for Nanonets' own model
Data
| Monthly downloads (dataset) / stars (repo) / downloads (model) | |
|---|---|
| TRACE benchmark dataset (Hugging Face) | 44 |
| TRACE evaluation repo (GitHub) | 39 |
| Nanonets OCR-3 model (Hugging Face) | 288,000 |
Watch the pull requests
The next reading is dated and public: submissions of new systems land as GitHub pull requests 7. No verified score far above 38 has been posted. Until one is, the defensible budget line for enterprise agents is that most multi-step process work fails on the industry's own test.
Deepdive
AI-generated from this story and its cited sources. Not investment advice.
