Back to blog
AI EthicsNovember 16, 2026 6 min read

Eighteen Detectors. The Best One Catches Fourteen in a Hundred.

AP
Angelo Pallanca
Digital Transformation & AI Governance

TL;DR

  • A benchmark posted on 11 November 2026 ran 18 AI-video detectors over 10.000 clips, roughly 70% manipulated, recompressed the way WhatsApp, TikTok, Instagram, X, WeChat and RedNote recompress anything that gets reposted.
  • At a 1% false-positive rate, the best detector flags 14.1% of the fakes. The detector with the highest AUC, 0.693, flags 1.3%, so the two rankings disagree.
  • One strong detector falls from 0.93 AUC to roughly coin-flip within five re-encodes, then stays there out to thirty. Sending a file to someone is the attack.
  • Article 50(2) of the AI Act has required machine-readable marking since 2 August 2026, qualified by "as far as this is technically feasible" and by "the generally acknowledged state of the art". Research keeps pushing that state of the art toward the exemption.

Assumptions

  1. The video numbers come from a preprint, arXiv:2611.14941, by a group at Arizona State University with one co-author at NYU. They are the authors' own measurements on their own benchmark, not peer-reviewed and not replicated. I treat them as one careful measurement rather than a settled fact.
  2. The watermark numbers come from a second preprint, arXiv:2607.16010 of 17 July 2026, which tested open implementations of KGW, Unigram and SynthID-Text through the MarkLLM toolkit rather than Google's production system. Weaker evidence, used as indicative.
  3. The legal reading is mine. The Article 50 wording and the Code of Practice dates are documented, but the argument about where the escape clause leads is an argument.

Goal

Three things get sold as one product: marking synthetic content when it is born, detecting it afterwards, and the legal duty to do either. In Europe all three are now live, and only the third has a date on it.


The number nobody prints

The benchmark covers 18 detectors, 11 classic and 7 built on vision-language models, over 10.000 videos assembled from HowTo100M, OpenVid-HD and a crawl of socially fragile footage that makes up 53% of the set. Generators include Wan2.2, Hunyuan-1.5 and LTX-2, plus character replacement and relighting passes.

On the headline metrics the field looks mediocre but workable. D3 tops the group at 0.693 AUC and 0.642 Macro F1.

Nobody deploys at the AUC. You deploy at a false-positive rate you can live with, because every false alarm is a real video publicly called fake. Fix that rate at 1% and D3 catches 1.3% of the fakes. WaveRep, which looks worse on the summary metric, catches 14.1%, and the ordering flips.

AUC is the number in the press release. True positives at a survivable false-positive rate is the number that decides whether a review queue is worth staffing.

A buyer comparing vendors on accuracy gets a ranking with no relationship to the thing being bought.


What five reposts do

WaveRep scores 0.93 AUC on the clean version of the benchmark. Within five re-encodes it sits near 0.5, and it stays flat out to thirty. Half is a coin.

The authors separate two causes, which matters. New generators and unfamiliar content already cost accuracy before anything is compressed: one vision-language detector slides from 0.74 AUC on an older benchmark to 0.63 on this one, untouched. Then platform recompression wipes out the low-level traces that artifact-based detectors read. D3 holds around 0.69 through every condition, which is the stability of a detector that was never reading those traces to begin with.

So the thing that destroys the evidence is the ordinary act of sharing. The authors conclude that "no current design is robust to all of them", listing generator shift, re-encoding, camera effects and fragile content.


The clause that does the work

Article 50(2) has applied since 2 August 2026, extended to 2 December 2026 for systems already on the market. Mark the output in a machine-readable format, make it detectable as artificially generated, with solutions that are effective, interoperable, robust and reliable, "as far as this is technically feasible", taking into account "the generally acknowledged state of the art". The Transparency Code of Practice arrived on 10 June 2026, was found adequate by the Commission on 8 July and by the AI Board on 9 July, and the guidelines followed on 20 July. Measure 3.4 of that Code asks for an interoperability solution for detection mechanisms by 2 February 2027. The Code is voluntary.

Now put the second preprint next to it. A meaning-preserving paraphrase stripped the mark from 100% of the KGW texts, 100% of Unigram and 98.3% of SynthID-Text, counting only the texts the detectors had recognised in the first place, which they failed to do between 70% and 83% of the time on untouched watermarked output. The authors score no method above two of the five Daubert factors.

Every rigorous study of how marks survive strengthens the case that marking which holds up is not technically feasible, and technical feasibility is the condition the whole duty hangs on. The better the evidence, the lighter the obligation, and compliance drifts toward documenting the attempt.


A century of studying a machine that never worked

On 8 October 2002 the National Research Council published its review of the polygraph. Weak scientific basis after nearly a century of study, accuracy too low for security screening, and a specific reason: a test tuned to produce few false alarms misses most real violators, while the screened population is almost entirely innocent. The committee added that the polygraph still extracts confessions, from its reputation rather than its accuracy. Stephen Fienberg put it plainly: "National security is too important to be left to such a blunt instrument."

Agencies kept using it for decades afterwards. That is the part worth borrowing, because a detector's usefulness to an institution has never depended on the detector working.


Why this matters for your business

Four questions, and three of them cost nothing.

Ask a detection vendor for true positives at 1% and at 0.1% false positives, measured on reposted content, and refuse accuracy and AUC as answers. Ask what the curve does after five platform re-encodes. Decide in advance who absorbs the cost of a false positive, because a real recording publicly labelled fake is the error that ends up in front of a lawyer. And keep provenance in your own custody, signed at capture and held in your own archive, instead of expecting to read it back off a public feed.

By 2 February 2027 Europe will plausibly have an interoperability standard for reading a mark that most circulating copies no longer carry. The fight then moves to whose job it was to keep the original.

Want to discuss this further?

Request a proposal