AI Agent Benchmarks Are Finally Getting Real

TL;DR

• Zapier’s automation benchmark covers 657 tasks across 40 simulated SaaS apps and scores both task objectives and guardrails such as company policy. (source video QBtsPdN7Efc, 01:16; source video QBtsPdN7Efc, 01:23)
• Ron reports Fable 5 leading at 48.6%, only 0.1 percentage point above Opus 4.8, while Gemini 3.5 Flash stands out for its mix of objectives, guardrail violations, and cost efficiency. (source video QBtsPdN7Efc, 01:38; source video QBtsPdN7Efc, 02:13)
• The recorded open-weight result is much weaker: GLM 5.2 Max is the highest listed open model at 27.8%. Treat that as a dated result, not a permanent judgment on open models. (source video QBtsPdN7Efc, 02:22; source video QBtsPdN7Efc, 02:30)
• A single scalar score—one number that collapses every capability—is not enough. Read performance by domain and attach cost per task. (source video QBtsPdN7Efc, 02:44; source video QBtsPdN7Efc, 03:43)
• Memory is part of agent infrastructure: the video describes ATMA, Recontext, and block search as inference-time attempts to handle stale facts, evidence use, and long-context retrieval. (source video QBtsPdN7Efc, 04:11; source video QBtsPdN7Efc, 05:10; source video QBtsPdN7Efc, 05:28)

Ron’s verdict

Progress is not permission to trust an agent. A leaderboard becomes useful only when it resembles the job: finish the task, obey the business rules, stay inside the budget, and retain the right context over time. The closed models in this recording lead the open field, yet Ron says even the winner misses basic guardrails about half the time. That is prototype territory, not unattended production. Use these benchmarks to expose failure modes, then test the same constraints on your own workflow. (source video QBtsPdN7Efc, 02:00; source video QBtsPdN7Efc, 06:20; source video QBtsPdN7Efc, 06:34)

Key moments

• 00:00 — Why simple completion tests are not enough: Ron separates finishing one task from obeying business rules, controlling API spend, and coordinating projects. (source video QBtsPdN7Efc, 00:10)
• 01:00 — A benchmark for real workflows: Artificial Analysis’s Zapier leaderboard is introduced. (source video QBtsPdN7Efc, 01:00)
• 01:36 — The sobering leaderboard: the leading closed-model scores remain below 50%. (source video QBtsPdN7Efc, 01:36)
• 02:44 — Replace one score with domain evidence: six capability indices split performance across different kinds of work. (source video QBtsPdN7Efc, 02:44)
• 03:43 — Put cost beside capability: Ron argues that benchmark results need a per-task burn rate. (source video QBtsPdN7Efc, 03:43)
• 04:11 — Memory becomes the bottleneck: stale and current facts can collide in persistent assistants. (source video QBtsPdN7Efc, 04:11)
• 05:10 — Recontext and block search: long-context behavior is treated as an inference and retrieval problem. (source video QBtsPdN7Efc, 05:10)
• 05:57 — The operational through line: realistic constraints, cost-aware scoring, and engineered memory come together. (source video QBtsPdN7Efc, 05:57)

Useful quotes

“Even the best agent in the world, Fable 5, fails basic guard rails about half the time.” — Ron, source video QBtsPdN7Efc, 02:08

“you cannot compare agents in a vacuum anymore you need to know the burn rate for every workflow” — Ron, source video QBtsPdN7Efc, 03:58

“We’re not waiting for the next pre-training run to fix agent memory.” — Ron, source video QBtsPdN7Efc, 05:41

“And if you’re not optimizing for cost per task and inference time memory, you’re not really building an agent. You’re still building like a prototype or like a demo, right?” — Ron, source video QBtsPdN7Efc, 06:32

What the benchmark result actually says

The important design choice is the split between objectives and guardrails. An agent can complete a requested action while still violating the rule that made the action safe for a business. Ron compares this to an employee review: did the person finish the job, and did they follow company policy while doing it? (source video QBtsPdN7Efc, 01:23; source video QBtsPdN7Efc, 01:28)

The reported leaderboard is close at the top: Fable 5 scores 48.6%, Opus 4.8 scores 48.5%, Gemini 3.5 Flash scores 42.6%, and GBD 5.5x high scores 42.1%. Ron’s point is not that the 48.6% model is ready. He says the expected standard should be around 70% or 80%, while all the leaders still violate business rules. (source video QBtsPdN7Efc, 01:38; source video QBtsPdN7Efc, 01:49; source video QBtsPdN7Efc, 02:00; source video QBtsPdN7Efc, 02:05)

Gemini 3.5 Flash is the cost-aware outlier in Ron’s reading because it looks strong on objective per guardrail violation and cost efficiency. The transcript does not provide its per-task price, so this companion does not invent one. (source video QBtsPdN7Efc, 02:13)

The open-weight gap is also a snapshot with a narrow meaning. Ron reports GLM 5.2 Max at 27.8% as the highest listed open model and says the open ecosystem is not close to the closed frontier on these end-to-end, policy-bound tasks. That supports caution for this workload at recording time; it does not establish that every open model is bad at every job. (source video QBtsPdN7Efc, 02:22; source video QBtsPdN7Efc, 06:25)

Read an agent benchmark like an operator

This decision table is companion guidance derived from the problems Ron identifies. It is not an official scoring rubric from Artificial Analysis or Zapier.

Evidence to capture	Question before deployment	Why it changes the decision
Objective result and guardrail result	Did the agent finish without breaking policy?	Completion alone hides business-rule failures. (source video QBtsPdN7Efc, 01:23)
Domain-specific score	Does the score match finance, legal, healthcare, strategy and ops, engineering, or economics work?	Ron reports that rankings reshuffle by domain. (source video QBtsPdN7Efc, 02:51; source video QBtsPdN7Efc, 03:16)
Cost per completed task	What did a successful run actually burn?	Price-to-performance is steep, and narrow capability can make an expensive model worth using only in specific contexts. (source video QBtsPdN7Efc, 03:26; source video QBtsPdN7Efc, 03:58)
Persistent-memory behavior	Can the agent distinguish stale facts from current ones after months of use?	Ron describes “ghost memory” as retrieving conflicting old and current facts together. (source video QBtsPdN7Efc, 04:13; source video QBtsPdN7Efc, 04:19)
Your own guardrail failures	Which policies fail in the workflow you actually run?	Ron asks production users to report the guardrails they hit; the benchmark is a starting point for that local test. (source video QBtsPdN7Efc, 06:42)

The routing decision follows from those columns. Do not pick the overall winner and send it every task. First identify the domain and required policies; then compare completed-task cost and memory behavior. That workflow is companion analysis based on Ron’s cost, domain, guardrail, and memory framing—not a result measured in the video.

Memory is not a side issue

Ron connects benchmark quality to what happens after an agent has been running for months. Ghost memory is the video’s name for stale and current facts being retrieved together without the model reliably separating them. Ron reports that adding ATMA to graphidi improved conflict accuracy by 0.24 absolute on the LTP benchmark. The workpack contains no independent reproduction of that result. (source video QBtsPdN7Efc, 04:13; source video QBtsPdN7Efc, 04:28; source video QBtsPdN7Efc, 04:44)

The other two examples work at inference time rather than requiring a new pretraining run. Recontext is described as a training-free long-context inference harness that replaces model-internal evidence before answer generation and improved evidence use across eight 128K datasets. Block search is described as pushing million-token in-context retrieval. (source video QBtsPdN7Efc, 05:10; source video QBtsPdN7Efc, 05:24; source video QBtsPdN7Efc, 05:28)

The durable takeaway is architectural: evaluation must cover the model plus retrieval, context handling, cost, and policy adherence. Better base-model scores do not remove those system responsibilities. (source video QBtsPdN7Efc, 05:36; source video QBtsPdN7Efc, 05:57)

What changed since this video

The immutable workpack records this video as published on July 7, 2026; this companion was source-checked against its full transcript and all 345 timestamp segments on July 18, 2026. No external leaderboard, paper, repository, pricing page, or product documentation was added. The model names, exact scores, six indices, ATMA result, Recontext result, and block-search capability above are therefore a dated record of the video—not confirmation of their current state. Verify the latest primary materials before making a production or purchasing decision.

Related

• Perplexity WANDR: evidence-backed research-agent evaluation
• What an AI model tier list can and cannot tell you
• Frontier vs open-weight: choose by workload