Perplexity WANDR Changed the Research Game.. (No more AI slop)

TL;DR

• WANDR evaluates research agents against the live web and checks whether cited sources support the claims made from them.
• Ron reports 500 de-identified production-style tasks and 170,000 source-backed records across difficulty tiers.
• Research quality depends on the complete stack: model, tools, harness, search strategy, context, routing, loops, and final verification.
• A good model router must be stable under paraphrasing, not merely cheap and accurate on one phrasing.
• Automated evals should surface patterns at scale; domain experts still decide whether an answer is useful, prioritized correctly, and based on the right interpretation.

Ron’s verdict

This is a more useful direction than another static leaderboard. Real research does not have one permanent answer key: pages change, sources disappear, and two well-supported answers can reach the same conclusion through different evidence. WANDR’s important move is to re-fetch what the agent cited and test claim support. That still does not make an agent automatically trustworthy. It gives builders a better instrument for finding where trust breaks—and a reason to evaluate the entire research workflow rather than celebrating the base model.

Key moments

• 00:00 — Can you trust a research agent?: Ron opens with the correct default—important answers still need skepticism.
• 01:04 — What WANDR measures: the benchmark’s production-style tasks and source-support evaluation are introduced.
• 03:19 — Systems, not isolated models: agent performance is broken into harness, tools, prompts, search, routing, loops, and verification.
• 04:40 — The model-router stability test: equivalent prompts should not bounce unpredictably between experts.
• 05:33 — Automated evaluation still needs experts: scalable trace inspection and human judgment solve different parts of the quality problem.
• 07:17 — SlopCode-Bench and accumulated damage: sequential tasks reveal technical debt that one-ticket tests miss.

Useful quotes

“Can I trust this answer? Of course, you shouldn’t. You should take it with a grain of salt.” — Ron, source video N2RttyY5HdM, 00:03

“what Wander does is it re-fetches the sources that an agent cites and checks whether its claims are actually supported by the underlying evidence.” — Ron, source video N2RttyY5HdM, 01:54

“automated evals are useful but not a replacement for domain expertise.” — Ron, source video N2RttyY5HdM, 05:34

“The question is not only did it close the ticket, it is more so what did it leave behind.” — Ron, source video N2RttyY5HdM, 08:01

The details

Why a fixed answer key breaks on live research

Static references work well when a problem has one stable answer. Web research is messier: pages change, cited material disappears, and a correct conclusion may be supported by evidence different from the benchmark’s original reference (01:31). WANDR’s response, as described in the video, is to re-fetch an agent’s cited sources and evaluate whether the underlying evidence actually supports its claims (01:54).

Ron reports that the benchmark contains 500 tasks built from de-identified production research work and 170,000 source-backed records across difficulty tiers (01:04). He also describes the same production-style traces as a possible environment for reinforcement learning: training and evaluation can resemble work people are actually trying to automate, rather than synthetic exercises chosen because they are easy to grade (02:36).

The video says Ron cloned the repository and asked Hermes to set up the pipeline. At recording time, that setup required an OpenAI API key and a Perplexity API key (03:00). This companion does not provide setup commands because none appear in the transcript.

Evaluate the full agent stack

Once an agent is researching the live web, its result depends on more than the underlying model. Ron lists the harness, tools, prompt, search strategy, model-selection logic, loop, and final verification as parts of the system (03:19). That changes the useful question from “Which model scored highest?” to “Which complete configuration produces supported answers at an acceptable cost?”

The video cites research that reduced agent-system cost by 89% while matching 100% of the best static configuration. Ron’s emphasis is that the gain came from full-system configuration, not merely sending easy prompts to a cheap model and hard prompts to an expensive one (03:42). Those numbers are reported from the video and are not independently reproduced here.

A router should survive paraphrasing

Ron highlights a warning from Google DeepMind’s related work: accuracy and cost do not fully describe a multi-model router. The router should make meaningful distinctions between its experts and remain stable when the same request is rephrased (04:40). If equivalent requests go to completely different models, the routing rule may be responding to wording noise rather than a real task difference.

That instability matters operationally. A routing decision that changes under harmless paraphrasing becomes difficult to optimize, reproduce, and debug (05:06). A practical router eval should therefore include groups of equivalent prompts and inspect both their outputs and selected experts.

A research-output audit you can use

This is companion analysis derived from the evaluation problems in the video. It is not presented as WANDR’s official scoring rubric.

Check	Question to ask	Failure signal
Claim coverage	Does every important factual claim have a source?	Unsourced numbers, dates, comparisons, or recommendations
Entailment	Does the cited passage actually support the claim?	A reputable link that discusses the topic but not the assertion
Freshness	Was the source fetched recently enough for the question?	Current-state claims based on stale or missing pages
Source quality	Is the source primary and appropriate to the claim?	A roundup cited instead of the original data or documentation
Router stability	Do paraphrases reach sensible, consistent experts?	Equivalent tasks routed unpredictably
Human usefulness	Is the answer correctly prioritized for the decision?	Rubric-compliant output that buries the consequential finding

Run automated checks across many traces to find recurring gaps, then give a domain expert the questionable claims and source passages—not only an aggregate score. That matches Ron’s proposed division of labor: eval agents surface patterns; experts decide what good work means (05:43).

Long-run quality is a separate benchmark

The same problem appears in coding. A one-ticket benchmark says little about the tenth change in the same repository. Ron uses MiniSWE-Agent to show that benchmark infrastructure and task packaging affect what a model score means (06:46). He then points to SlopCode-Bench, which looks for accumulated duplication, brittle abstractions, unnecessary dependencies, and work that makes the next handoff harder (07:17). The operator question is not just whether the agent finished. It is what condition the system was left in.

What changed since this video

The video was published on July 15, 2026, and this companion was source-checked on July 17, 2026. It reports WANDR’s size, behavior, access requirements, and related research exactly as described in the transcript. No external freshness check was included in the immutable workpack, so this article does not claim that repository setup, API-key requirements, benchmark data, or reported results remain unchanged. Verify those items against the current official project materials before running or citing the benchmark.

Related

• Lesson: What is an AI model tier list? — how to read model evidence without mistaking one score for a complete decision.
• Lesson: Frontier vs open-weight decisions — evaluating the deployment system around model capability.
• Lesson: Coding with Claude Code — verification and handoff practices for coding agents.
• Watch the source video on YouTube.