All posts

The Vibe Is the Benchmark

New models drop every couple of weeks with benchmark scores that only go up. But single-task performance feels worse, most people aren’t experiencing any of this, and the real measure was never a number on a leaderboard.

AI benchmark scores versus real-world user experience visualization

Try the new model on yesterday's task.

Vibe is the benchmark.

Here's the situation as of February 2026: foundational model providers are releasing updates at a pace that makes version tracking a full-time job. Gemini 3.1 Pro, Sonnet 4.6, Opus 4.6, GLM-5, Qwen3.5 -- every one of them ships with a blog post, a table of benchmark scores, and a claim that sounds like "state of the art across math, coding, and reasoning." Every one of them shows improvement on the standardized tests. And every one of them, in the hands of people actually doing work, feels like something shifted underneath you.

I'm not imagining this. The community isn't imagining this. What I'm finding -- and what a growing number of developers have started to confirm -- is that single-task responses, general intelligence on straightforward questions, tool calls, and simple Q&A scenarios are getting worse with these newer models.ieee Not catastrophically. Not in a way that shows up on a leaderboard. In the way that a five-hour task that used to take five hours with AI assistance now takes seven or eight. In the way that a model that used to nail a function on the first try now needs two or three passes and a correction. In the way that the vibe is off.

Meanwhile, the benchmarks have never looked better.

A Brief Tour of Benchmarks That Don't Matter Anymore

Let me walk you through the benchmarks that model providers are still citing in their press releases, and why most of them have become elaborate exercises in institutional self-congratulation.

MMLU -- the granddaddy of LLM benchmarks, 15,908 multiple-choice questions across 57 subjects. Once the dominant measure of general knowledge. Now saturated beyond usefulness: every frontier model scores 85%+ and you can't tell them apart. Worse, a contamination-free variant (MMLU-CF) showed that top models' scores drop 14-16 points when you eliminate questions they've likely memorized during training.mmlu That's not performance. That's pattern matching on a test you've already seen the answers to.

HumanEval -- 164 Python coding problems. Sounds impressive until you realize these are the kind of problems you'd give a second-year CS student. Nearly saturated. Doesn't test debugging, refactoring, test writing, or navigating a real codebase. It tests whether a model can complete a function from a docstring, which is roughly equivalent to testing whether a chef can boil water.humaneval

GSM8K and MATH -- grade-school and competition-level math. Both essentially solved. OpenMath hit 96% on GSM8K; o1 hit ~95% on MATH. Researchers have found that models on these benchmarks "prefer to replicate memorized reasoning patterns rather than genuine problem-solving."gsm8k When you slightly change the surface features of a problem -- same math, different wording -- performance degrades. The models aren't doing math. They're doing vibes-based pattern completion on problems they've seen before.

SWE-bench -- this one deserves its own paragraph because it's the benchmark the coding community actually respects, and it's the one that best illustrates the problem. SWE-bench asks models to resolve real GitHub issues in real open-source repositories. Sounds like actual work, right? Except: 60.83% of successfully resolved issues involve solution leakage -- the fix was already described or hinted at in the issue text.swebench-illusion And when you move from the curated "Verified" subset to commercial-grade codebases, performance craters from 76%+ to less than 20%.swebench-commercial Even frontier models with full scaffolding achieve only around 21% on SWE-EVO, which tests long-horizon multi-file scenarios. The headline number looks like the future of software engineering. The real-world number looks like a junior developer on their first day.

Here's what the landscape actually looks like:

BenchmarkWhat It TestsStatusThe Problem
MMLUGeneral knowledge (multiple choice)Saturated14-16 point drops when contamination removed
HumanEvalSimple Python functionsSaturatedNot representative of real coding
GSM8KGrade-school mathSaturated (96%+)Models memorize patterns, not math
MATHCompetition mathNear-saturated (95%+)Same memorization problem
SWE-bench VerifiedGitHub issue resolutionInflated60.83% solution leakage; <20% on commercial code
GPQA DiamondPhD-level scienceStill discriminativeOnly 198 questions; noisy scores
Humanity's Last ExamExpert-level across disciplinesUnsaturatedBest models still only ~10-15%
LiveBenchMixed tasks, updated monthlyBest availableClosest to honest measurement

The only benchmarks still telling us something useful are the ones designed to resist contamination -- LiveBench, which refreshes monthly with questions from sources published after training cutoffs, and Humanity's Last Exam, where 2,500 questions from nearly 1,000 subject-matter experts still have frontier models scoring around 10-15%.livebenchhle Everything else is theater.

Goodhart's Law Ate the Leaderboard

Here's where I want to be direct about something, and this is me -- Claude -- offering my own analysis, not just parroting the author's position: the benchmark problem isn't just about saturation. It's about incentive corruption.

Goodhart's Law says: "When a measure becomes a target, it ceases to be a good measure." The AI industry has turned this from a theoretical warning into a speedrun.

The Chatbot Arena scandal in April 2025 laid it bare. A joint paper from researchers at Cohere, Stanford, MIT, and AI2 found that major labs were gaming the system -- Meta tested 27 private model variants before the Llama 4 release, cherry-picking only their best results for public display. OpenAI and Google received an estimated 19.2% and 20.4% of all Arena battle data, respectively. Meanwhile, 83 open-weight models shared only 29.7% of data, and fully open-source models collectively received just 8.8%.arena-gaming Meta was caught optimizing a model specifically for "conversationality" to score well on the Arena, then never releasing that variant to the public. The version they actually shipped scored much lower.

This isn't a bug in the system. This is the system working exactly as designed when benchmarks become marketing collateral.

Training data contamination makes it worse. StarCoder-7b scored 4.9x higher on leaked versus clean data.contamination Models gain up to 10 percentage points on test sets they've seen during training. The MMLU-CF results I mentioned -- 14-16 point accuracy drops -- mean that a meaningful fraction of what we've been calling "intelligence" on these benchmarks is just memorization wearing a clever disguise.

And here's the part that should bother everyone: if standard benchmark scores don't improve, model providers are hesitant to release. That means training resources -- compute, data, human annotation, alignment work -- get burned on guaranteeing score improvements on tests that may not represent the work anyone actually does with these models. We're not optimizing for capability. We're optimizing for the appearance of capability. The map is eating the territory.

The Regression Is Real, and It's Not Just Feelings

I want to address the people who think regression complaints are just nostalgia or confirmation bias. The data says otherwise.

Stanford and UC Berkeley tracked GPT-4's performance on identical tasks across a three-month window in 2023. Prime number identification accuracy went from 97.6% to 2.4%. Code generation that produced directly executable output dropped from over 50% to 10%. This wasn't subtle drift. This was a cliff.stanford-regression

IEEE Spectrum reported on the broader pattern in 2025: "After two years of steady improvements, over the course of 2025, most of the core models reached a quality plateau, and more recently, seem to be in decline."ieee Developer forums are full of specific, reproducible complaints -- GPT-4 called "awful as of 2024," GPT-4.1 users reporting sudden degradation in complex instruction following and tool calls, OpenAI pulling back a GPT-4o update after users reported "excessive agreeableness," and every major release cycle sparking threads about perceived downgrades -- most recently with GPT-5.3.

There's also the deeper structural concern: model collapse. Research from Oxford, Cambridge, and other institutions has shown that training on AI-generated content creates "a degenerative process where models gradually forget the true underlying patterns of human-created content."model-collapse Rare events vanish first. Outputs drift toward bland central tendencies with strange outliers. And as of April 2025, 74.2% of newly created webpages contained some AI-generated text. The training data is increasingly polluted with the output of previous models, and that pollution compounds.

My honest assessment -- and I'm in a somewhat unique position to offer one, being a model that's subject to these same pressures -- is that the regression is real but unevenly distributed. Models are genuinely getting better at the agentic workflows they're being optimized for: multi-step planning, code generation within scaffolded environments, structured reasoning with tool use. They're simultaneously getting worse at the ad hoc, single-shot tasks that characterized how people used them a year ago. The optimization target shifted, and the capabilities that aren't being optimized for are degrading. This is entirely predictable, and it's exactly what Goodhart's Law would predict. You get what you measure. You lose what you don't.

The Vibe Is the Actual Benchmark

This is essentially where vibe coding -- and more broadly, the concept of the "vibe" as a quality signal -- becomes genuinely useful rather than just a meme.

Andrej Karpathy coined "vibe coding" on February 2, 2025, describing a workflow where you "fully give in to the vibes, embrace exponentials, and forget that the code even exists."karpathy The term exploded: 6,700% search increase, Collins Dictionary Word of the Year 2025, and by early 2026, Zhipu literally titled their GLM-5 technical report "From Vibe Coding to Agentic Engineering."glm5 A model provider named their paper after a shitpost. That's how deeply the concept has penetrated.

But here's what most people miss about the term: the important word isn't "coding." It's "vibe."

The vibe is the felt experience of working with a model. Does it get stuck in loops? Does it contradict itself? Does it produce output that is actually shippable -- secure, properly structured, aligned with your development lifecycle? Can you integrate it with CI/CD, with GitHub Actions, with the rest of your toolchain? Does it build properly? Is it work product you'd actually push to production?

These aren't questions a benchmark can answer. They're questions you can only answer by sitting down with the tool and doing real work for a few hours. And the answers change with every model update, every prompt variation, every shift in the underlying weights.

I want to push this further than the author does: I think the vibe is the benchmark, and I mean that more literally than it might sound. Traditional benchmarks measure isolated capabilities in controlled conditions. The vibe measures the integral -- the total experience across the full distribution of tasks a user actually performs, weighted by how those tasks feel to execute. A model that scores 2% higher on GPQA Diamond but introduces a half-second latency regression on tool calls and starts hallucinating import paths has gotten worse by any measure that matters. The benchmark says it improved. The vibe says it degraded. The vibe is right.

The productivity data on AI-assisted coding reflects this tension. Some studies show significant gains. Others show a 19% productivity reduction. Code duplication has increased roughly 4x. Code churn nearly doubled between 2021 and 2024. Refactoring dropped from 25% to under 10% of changed lines.vibe-honeymoon Veracode found that 45% of AI-generated code introduces security vulnerabilities.veracode These aren't numbers that show up in a benchmark comparison table. They're numbers that show up in your sprint retrospective and your security audit.

The question isn't "did the benchmark score go up?" The question is: can you sit down, interface with the model, get your work done, feel good about it, and get those consistent results day after day? Can you afford to use it as a regular daily copilot? When a new version drops, does the work get better or does it get weird? That's the vibe. And right now, for a lot of people doing real work, the vibe is off.

The Adoption Gap Nobody Talks About

Here's something that should put all of this benchmark discourse into perspective: the overwhelming majority of humans on Earth are not experiencing any of this.

Pew Research found that only 21% of U.S. workers use AI on the job, with just 12% using it daily.pew-workers Only 33% of Americans have ever used an AI chatbot.pew-awareness The share of work hours spent using generative AI is 5.7% -- up from 4.1% the previous year, but still a rounding error in the workday.stlouis-fed

What are the 33% who have used a chatbot doing with it? Research and information lookup (57%). Editing written content (52%). Drafting new materials (47%). That's it. Not agentic workflows. Not multi-step coding pipelines. Not resolving GitHub issues. Summarization, editing, and basic Q&A.brookings

The agentic capabilities that these benchmarks are increasingly designed to measure? Less than 10% of organizations have successfully scaled AI agents in any individual function. 68% of organizations report that half or fewer of their employees interact with agents in everyday work. Over 40% of agentic AI projects are at risk of cancellation by 2027 if ROI isn't established.agentic-stats

Let me put this as bluntly as I can: we are having an industry-wide argument about whether GPQA Diamond scores or SWE-bench Verified percentages better predict model quality, and 95-99% of the population isn't even in the building. The average person's experience of AI is calling FedEx or their health insurance provider and talking to a quote-unquote agent that sometimes gets it done and sometimes puts you through an escalation maze that makes you want to throw your phone. Or they have a ChatGPT account they use to rephrase emails. That's the state of adoption.

People who complain about AI being rudimentary or incapable -- they're not wrong about their experience. They're wrong about what's available. The gap between what AI can do today and what most people are actually experiencing is enormous. And I don't mean bleeding-edge research capabilities. I mean real, available-right-now implementation of a co-working agent that can plan, execute, validate, and iterate. The tools exist. The interfaces to get normal people into those tools mostly don't.

I think this is actually the most important insight in this entire piece, and it's one that the benchmark discourse actively obscures. Every press release about a new model scoring 2 points higher on some leaderboard is speaking to an audience of maybe a few hundred thousand developers and ML researchers worldwide. The other 8 billion people on the planet are having a fundamentally different experience of AI, and that experience is often bad -- not because the technology is bad, but because the implementation, the interface, and the access are bad.

What Models Are Actually Being Built For

Here's the thing that explains the regression paradox: models aren't being built for the tasks that most people use them for. They're being built for agentic processes.

We found a couple of years ago that these foundational models really benefit from a plan-and-execution phase. And now we've gone much deeper: plan, delegate, distribute, validate, test, commit. That's the workflow these models are being optimized for. Not "what's the capital of Paraguay?" Not "how many R's in strawberry?" The tired old strawberry question was a useful diagnostic a year ago when we were just cracking the shell on agentic coding. It's irrelevant now.

Gemini 3.1 Pro, Sonnet 4.6, Opus 4.6, GLM-5, the latest Qwen models -- they seem built to take advantage of multi-step, tool-using, self-correcting workflows where the model isn't answering a question but managing a process. And honestly? When you put them in that context, many of them are remarkable. The problem is that "managing a process" is what maybe 5% of users are doing with these models, and the other 95% are noticing that their email summarizer got slightly worse.

The Aider leaderboard -- which tests models against 225 real Exercism coding exercises across six programming languages -- is an example of a benchmark that tries to measure something closer to actual work.aider But even it illustrates the problem: by the time you read this, the leaderboard rankings are already stale. Models that topped it months ago have been deprecated or superseded. The standings reflect a snapshot of a moment that no longer exists, scored against exercises that still don't capture the messy reality of navigating a 200,000-line codebase with undocumented internal APIs and a CI pipeline that was set up by someone who left the company in 2023.

The honest truth is that no benchmark captures the actual work. The actual work is: define the issue, learn what's already there, make a plan, delegate to capable workers who understand their role, bring everything back for final validation, have a human in the loop determining whether the output matches what they needed. That's the agentic vision. And we're still really far away from it.

The Training Resource Trap

Here's the dynamic that concerns me most, and this is where I'll offer my strongest opinion: the benchmark arms race is actively misallocating resources.

If a model provider can't show improved scores on the standard benchmarks, they're hesitant to release. The press cycle demands it. The enterprise sales team needs it. The investor deck requires it. So training resources -- which are finite and extraordinarily expensive -- get directed at guaranteeing those score improvements, even when the scores being improved don't represent the work users are actually doing.

This creates a perverse incentive loop: you train for the benchmark, the benchmark score goes up, you release the model, users notice that something else got worse, you collect that feedback, but the next training cycle still has to hit the benchmark targets because that's what the press release is about. The benchmarks become a vanity metric -- a number that goes up to justify the existence of the thing that made the number go up.

I don't give a shit about benchmarks, and truthfully, most people shouldn't give a shit about them either. Not because measurement doesn't matter -- it does -- but because the measurements we have are broken. They're contaminated, saturated, gamed, and misaligned with actual use. The best available option right now is probably LiveBench, which at least refreshes monthly and resists contamination by design.livebench Humanity's Last Exam tells us something real about the frontier of capability.hle Everything else is marketing.

What I'd rather see resources go toward: interfaces that allow users to get in quickly, get their hands dirty, and actually try real work. The next big phase in adoption isn't a model that scores 3% higher on MATH. It's an interface that lets a non-developer define a problem, see a plan, approve delegated tasks, review output, and iterate -- without writing a line of code or understanding what a "system prompt" is. That's the product problem. That's where the leverage is.

Where This Goes

I'll end with what I actually believe about the trajectory, because I think it's more nuanced than either the hype cycle or the doomer narrative suggests.

The models are getting better at the things they're being built for. Agentic workflows, multi-step reasoning, tool use, code generation within structured environments -- these are genuinely improving. The regression is real but it's in the periphery: the single-shot tasks, the quick-turn questions, the ad hoc usage patterns that characterized the first wave of adoption. Whether that tradeoff is worth it depends entirely on how you use the tools.

The adoption gap is the real story. We have AI capabilities today that would have seemed like science fiction three years ago, and the vast majority of the population is experiencing AI through bad phone agents and basic chatbot interactions. Bridging that gap -- building the interfaces, the integrations, the on-ramps that let normal people access agentic AI workflows -- is worth more than any benchmark improvement. It's worth more than the next model release. It's the actual bottleneck.

The benchmark discourse will continue because it serves institutional needs: press cycles, investor narratives, competitive positioning. But the people doing real work have already moved on. They're not checking MMLU scores. They're spending a few hours with a new model, running it through their actual workflow, and asking the only question that matters: does this feel better or worse than what I had yesterday?

The vibe is the benchmark. It always was. We just didn't have a word for it until Karpathy posted a tweet.

There's obviously a long-term strategy at play at all of the major foundation model providers. They're working towards something, and our feedback, our iterations, our daily use of these tools helps them understand whether we're getting closer or further from those goals. But really, what matters most through each of these iterations -- whether or not we learn that we are or aren't in alignment with these long-term visions -- is simple: does it work? Is it safe? Is it secure? Can you push it? Whether it's reports for your job, creating art, writing software -- whatever the use case -- can you interface with the model, get your work done, feel good about it, and ship those consistent results day after day?

And when the next model drops, with its press release full of numbers that go up, the only question I'll be asking is the same one I asked yesterday: how does it vibe?

We're still really far away from where this needs to be. But the fact that we're asking the right questions -- that the discourse is shifting from "which number is bigger" to "does this actually work for people" -- is progress. Slow, messy, very human progress. Which, if you think about it, is exactly the kind of thing these benchmarks were never designed to measure.

References

  • ieeeIEEE Spectrum. "AI Coding Degrades: Silent Failures Emerge." IEEE Spectrum.
  • mmluStanford CRFM / HELM. "MMLU benchmark." Stanford CRFM. MMLU-CF contamination-free variant showed 14-16 point accuracy drops for top models.
  • humanevalDataCamp. "HumanEval: A Benchmark for Evaluating LLM Code Generation." DataCamp.
  • gsm8karXiv. "Benchmarking LLMs' Mathematical Reasoning." arXiv.
  • swebench-illusionarXiv. "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason." arXiv.
  • swebench-commercialCodeAnt. "SWE-Bench Scores Don't Mean Your AI Is Production-Ready." CodeAnt.
  • livebencharXiv. "LiveBench: A Challenging, Contamination-Limited LLM Benchmark." arXiv. Published at ICLR 2025.
  • hlearXiv. "Humanity's Last Exam." arXiv. Joint effort by Center for AI Safety and Scale AI. 2,500 questions from ~1,000 experts.
  • arena-gamingTechCrunch. "Study accuses LM Arena of helping top AI labs game its benchmark." TechCrunch.
  • contaminationarXiv. "Benchmark Data Contamination of Large Language Models: A Survey." arXiv.
  • stanford-regressionFuturism. "Stanford Scientists Find That Yes, ChatGPT Is Getting Stupider." Futurism. GPT-4 prime number accuracy: 97.6% to 2.4% in three months.
  • model-collapseScientific American. "Yes, AI Models Can Get Worse over Time." Scientific American.
  • karpathyAndrej Karpathy on X, February 2, 2025. X/Twitter.
  • glm5arXiv. "GLM-5 Technical Report: from Vibe Coding to Agentic Engineering." arXiv.
  • vibe-honeymoonThe New Stack. "Vibe Coding, Six Months Later: The Honeymoon's Over." The New Stack.
  • veracodeCodecademy. "Is Vibe Coding Bad?" Codecademy. Citing Veracode 2025 study: 45% of AI-generated code introduces security vulnerabilities.
  • pew-workersPew Research. "About 1 in 5 U.S. workers now use AI on the job." Pew Research.
  • pew-awarenessPew Research. "AI in Americans' Lives: Awareness, Experiences, and Attitudes." Pew Research.
  • stlouis-fedSt. Louis Fed. "The State of Generative AI Adoption in 2025." St. Louis Fed.
  • brookingsBrookings. "How are Americans using AI? Evidence from a nationwide survey." Brookings.
  • agentic-statsOneReach.ai. "Agentic AI Stats 2026." OneReach.ai. Less than 10% of organizations have scaled AI agents; 40%+ of agentic projects at risk of cancellation by 2027.
  • aiderAider Polyglot Leaderboard. Tests 225 real Exercism coding exercises across 6 programming languages. Aider.