TL;DR
On 3 November 2026 a researcher at Shandong University audited two public agent leaderboards and found four errors in the measuring software, not in the agents (arXiv 2611.04399). One board graded one agent in 98 percent of its episodes and its rival in 51 percent, counted the ungraded half as failures, and so produced a thirteen point gap that vanished when those episodes were graded. My view: the agent market is priced on numbers produced by code nobody audits, and the check that catches the worst of it is free. Before comparing two scores, ask how many runs were scored.
Starting assumption: HAL's maintainers reproduced all three issues reported to them and posted dated disclosures in about five weeks, so this is about unaudited instruments, not bad faith. Goal: leave with four questions for your next model evaluation, three of which cost nothing to answer.
What the audit found
The thirteen points made the title. They were the least interesting part.
Cost was doubled. A wrapper span and its child carried the same usage and the aggregator added both: exactly twice in 70 of 134 runs, and 36.5 percent above the real figure overall.
The grader wrote in the agent's log. The gold replay used for scoring was recorded as the agent's own actions in 68 percent of 2,334 archived records. In 58 of them the agent called no tool, yet the trace shows the full correct sequence.
Re-grading moved the board. An announced fix reversed 4 of 136 pairwise orderings. On another benchmark a file check misread four entries and their scores fell by 59 to 82 points, with no error shown. The command, as the paper puts it, "prints a number."
Why the number survives anyway
Hubble launched in April 1990 with a mirror ground to the wrong shape. The rig that had certified it held a lens 1.3 mm out of position, so it confirmed the error it had introduced. Two other instruments in the same shop showed the aberration. Both were set aside because the first was labelled certified and they were not, a word the board found had never been defined anywhere.
An instrument becomes authoritative because it is the one everybody quotes. After that, disagreeing with it looks like a defect in the challenger.
Why this matters, and what to ask
Menlo Ventures put enterprise AI deployment spend at about 177 billion dollars by September 2026, with 86 percent of model API spend on two vendors chosen for capability over price. Capability, in practice, is a number off a leaderboard.
In April 2026 the EvalEval Coalition costed one HAL-style run at roughly $40,000, rising to about $320,000 with eight reruns per cell. Their line is that whoever pays for the evaluation writes the leaderboard. Fair, and here is what they left unsaid: none of these four errors needed a rerun. Three sat in the published artifacts, found by one person.
Four questions, then. What fraction of the runs was graded? Is cost taken from root spans, and what is the all-span to root ratio? Is every action tagged as agent, user simulator or grader? Which evaluator version produced this score, and where is the diff from the last re-grade?
The graded fraction is one column, and it decides whether you are reading a measurement or a number that prints.