How to Verify a Viral 'AI Can Now Do X' Claim

The AI space generates more hype claims per week than almost any other domain. 'AI can now pass the bar exam.' 'AI beats doctors at cancer diagnosis.' 'AI has achieved AGI.' Each of these circulates as a confident assertion — and each, on closer inspection, involves significant caveats, cherry-picked benchmarks, or misleading framing.

These claims matter because they influence investment decisions, hiring decisions, policy debates, and how non-technical people understand what AI actually can and can't do. When an AI capability claim spreads before the nuance catches up, the consequences range from bad product decisions to distorted public policy.

The irony is that asking an AI model whether an AI capability claim is true is genuinely tricky — models may be trained on the inflated headlines, may not have context on the benchmark conditions, or may simply lack the specific technical knowledge to evaluate the claim accurately.

Running an AI capability claim through five models is useful precisely because they have different training data, different relationships to benchmark literature, and different tendencies to flag speculative claims. When GPT-5.2, Claude Opus 4.5, Grok 4, Perplexity Pro, and Gemini 2.0 Flash all agree a claim is overstated — that's meaningful signal. When they split, the splits often reveal exactly where the nuance lies.