TL;DR
The best AI agent went from finishing 2.5 percent of real freelance projects last October to just over 21 percent this month, measured against the same 240 jobs. That headline travelled everywhere. The failure analysis printed underneath it did not, and it says something far more useful: when agents fail, they mostly fail at finishing and packaging, not at thinking. Which means the cheapest way to move your own number has very little to do with buying a smarter model, and the people cutting headcount in AI's name are reading the wrong instrument entirely.
Last October the best AI agent in the world could finish two and a half percent of real freelance projects. This month four different models clear seventeen percent and the leader sits at 21.25. Same 240 jobs, same scoring, roughly eight and a half times in under twelve months.
Everyone quoted that number. Almost nobody read the failure analysis a few pages later, which is where the useful part lives.
The number everyone quoted
The Remote Labor Index is a joint benchmark from Scale AI and the Center for AI Safety, published on 29 October 2025 as arXiv 2510.26787. It runs agents against 240 real paid projects across 23 domains, each one a job a human freelancer actually delivered. Median project: 11.5 hours and 200 dollars. In total, over 6,000 hours of human work worth more than 140,000 dollars.
The bar is what makes it worth reading. There is no multiple choice and no partial credit for a good attempt. Three human reviewers look at the deliverable and decide whether a real client would have accepted it. Agreement between reviewers runs at 94.4 percent.
At launch nothing cleared 2.5 percent. Manus led at 2.5, Grok 4 and Sonnet 4.5 at 2.1, Gemini 2.5 Pro at 0.83. Today the public leaderboard reads Opus 5.5 at 21.25, GPT 6 Astra at 20.83, Fable 5.1 at 17.92, Gemini Argon 4 at 17.08.
The number nobody quoted
The same work includes a breakdown of why the failures failed. Categories overlap, so they do not sum to a hundred.
Poor quality: 45.6 percent. Incomplete: 35.7. Corrupted files: 17.6. Inconsistencies across deliverables: 14.8.
Read the last three together and the picture changes. More than a third of failures stopped before the end. Nearly one in six arrived as a file the client could not open. Roughly one in seven contradicted itself across its own deliverables.
None of that is a thinking problem. It is a shipping problem.
The dock, not the ship
In April 1956 a converted tanker called the Ideal X left Port Newark for Houston with 58 containers welded to its deck. The ships were already fine. The routes were fine. The cargo was fine. What was broken was the dock, where up to half the cost of a sea shipment quietly sat in the work of loading and unloading it by hand.
Nobody fixed that with a faster ship. Marc Levinson's numbers in The Box: loading loose cargo ran about 5.83 dollars a ton, and loading the Ideal X came in under 16 cents.
Agent work is pre-container. The reasoning went up eight and a half times in a year. The dock is still manual.
Why this matters for your business
When I look at agent pipelines that stalled inside a company, the postmortem almost never says the model was not clever enough. It says the output came back in the wrong format, or stopped at the second of four deliverables, or three documents disagreed about the same figure. Three of the four failure modes in the index are testable with no model in the loop at all: does the file open, is every requested artifact present, do the numbers agree across them.
Then there is the expensive part. The Center for AI Safety also runs a wage loss dashboard, live since 29 July 2026. Its October update puts US job cuts that cited AI at more than 117,000 for 2026, and estimated annual wages lost at about 8.5 billion dollars, built from announced layoffs and state wage averages. CAIS says plainly that it does not verify AI caused any of them.
So one instrument counts accepted deliverables and the other counts press releases. The firms cutting staff in AI's name have never run the acceptance test on their own work. If they did, the honest sentence would not be "AI does 21 percent of jobs". It would be "AI finishes 21 percent of self-contained projects that arrive with a brief, a client and a deadline", which describes almost nothing inside an actual organisation.
The rate will keep climbing. When it reaches forty percent the story will again be written about a smarter model, and it will again be wrong. Somebody will have standardised the box. (My guess is that person will not be sitting in the AI team.)