Scale AI just published the benchmark that prices its own Qatar contracts
Scale AI's new Singapore legal benchmark measures exactly the gap that its Qatar contracts, signed three days earlier, pay Scale to close. Oracle's sovereign-AI story and private legal-AI valuations now have a number to argue with.
Vincent Jiangtwo-minute read
A test that prices the gap
On November 12, Scale AI$29B — Scale AI, private, latest valuation $29B and Singapore's Infocomm Media Development Authority (IMDA) published SG-LegalClarify, a benchmark putting 19 language models through 1,860 scenarios across ten areas of Singapore law, graded against 12,546 lawyer-written criteria 1. Twenty-three verified lawyers wrote the 1,100 source questions 1. The result is the first dated price tag on AI doing legal work, and it landed three days after Scale sold Qatar two more AI-for-justice contracts.
The score that should worry buyers
Even when models gathered every fact lawyers flagged as relevant, 55.6% reached the correct legal conclusion, 32.1% reached an incorrect one and 12.3% gave none 1. Ask a model a question missing a material fact and it stays silent about 39% of the time; when it does ask, it requests only 39% of the facts experts identified 1. Rule recall was the weakest skill, at 43.2% closed-book 1. Claude Fable 5 led closed-book at 69.8% of mandatory criteria; with web search, Kimi K3 led at 77.8% 1.
When models ask for every missing fact, 32.1% still reach the wrong legal answer
Data
| Value | |
|---|---|
| Correct conclusion | 55.6% |
| Incorrect conclusion | 32.1% |
| No conclusion | 12.3% |
What Scale sold while the ink dried
On November 10 at MWC Doha, Qatar's communications and justice ministries signed two agreements with Scale AI: SAK for notarisation and document authentication, Shoor for state litigation and legal research, under a five-year partnership targeting more than 50 government AI projects by 2029 2 3. Qatar's earlier Smart Legislative Advisor, built with the same three parties, aims to cut legislative review from about 35 days to five 3. On November 9, Scale and Oracle−2.86% — Oracle, down 2.86 percent today published a blueprint for keeping government AI workloads in-country, drawing on Scale's Qatar deployments 4.
The referee sells the game
One company now sells governments the AI workflow and the measurement of that workflow, and the benchmark prices the human-in-the-loop gap that closing is what Scale gets paid for 1 2. Scale is private; the listed read-through is Oracle (ORCL), whose sovereign-AI pitch leans on Scale's government record, plus private legal-AI marks that now have a benchmark to price seats against. Watch for a government buyer, not a vendor, publishing its own independent evaluation; that would break the loop. Deep dive in the sources below.
Get Model by email, free
The AI labs, their models, and the business behind them. In our daily email, every weekday.
Reader comments
0 comments