My thoughts on the future of agent evaluation

I started researching agent evaluation while my friend and I were working on Livin, a housing project with an agent for finding rental listings. Our evals would pass, but live tests kept revealing failures. We already had traces and an LLM reviewing them. What we struggled with was isolating what went wrong and knowing whether a change improved the agent.

The part I’m most interested in now is failures with serious consequences. An agent can book the wrong flight or delete important data, and an average success rate doesn’t tell us enough about those risks. The Holistic Agent Leaderboard paper says, “Agents with identical accuracy scores could exhibit vastly different behaviors and risk profiles for real-world use.” Its researchers observed incorrect credit-card use in flight-booking tasks. HAL, Section 4.2 OpenAI’s account of models seeking evaluation solutions through unauthorized access to infrastructure and third-party systems made that concern concrete for me. Hugging Face incident report I want evaluations to examine what an agent did to reach its answer, including the data and tools it used.

HAL offers a way to compare models and scaffolds across benchmarks under standardized evaluation conditions. The scaffold includes the prompts, tools, and control logic around the model, so changing the model is only one way to change the agent. Using Docent to analyze logs, the researchers found benchmark-answer lookups and scaffold data leakage. Verification and self-correction were also associated with higher task success, though that doesn’t establish causation. HAL, Sections 3–4 I’d like to use this kind of setup to test which changes help and understand why.

Those comparisons also need to account for cost, time, and consistency. HAL’s cost–accuracy Pareto frontiers help show whether a performance gain justifies the expense. For a voice agent, waiting time matters too. Emerging Trends in AI Agent Evaluation distinguishes $\mathrm{pass}@k$, success at least once in k attempts, from $\mathrm{pass}^{k}$, success across all k attempts. Section 1d I care about that repeated reliability, but it still depends on whether the grader checks for unacceptable behavior.

The finding that stayed with me most came from Agentic Self-Awareness: “recognition alone does not ensure an appropriate, agent-authored response.” One Terminal-Bench baseline recognized blockers on 84.4% of tasks but met the stricter behavioral criteria on only 40.6%, including non-fabrication and agent-authored deferral for the correct reason. Sections 2–3 I’ve also seen agents immediately agree when I challenge an answer. I want their confidence to follow the evidence, including when to stop or reconsider. The paper tests a single reconsideration challenge; that leaves repeated pressure an open question. Its harness-search gains also don’t establish reliable transfer to unseen blockers.

For personalized agents, some of that evidence comes from memory. Users shouldn’t need to repeat their preferences, but remembering them doesn’t guarantee the agent will act accordingly. MemPABench evaluates interaction preferences, including when to ask questions, seek confirmation, or act autonomously. In two examined cases, preferences were stored but never retrieved before the decision: “The bottleneck in both cases is retrieval, not storage.” Even when the correct preference was supplied and selected, agents sometimes failed to follow it. MemPABench, Findings 3 and 7 I want to understand where memory changes the agent’s trajectory and where that connection fails.

Replotted from MemPABench’s reported results. File memory improved overall preference selection and tracking of changes, while its mean context-sensitivity score was lower. These results cover four simulated users and four model backbones; the metrics use different scoring criteria, and uncertainty is not shown.

That’s the direction I want to explore through harnesses and evaluation, alongside AI safety and interpretability. Traces give us evidence about behavior, but they don’t fully explain a model’s internal computation, and automated trace evaluators need validation. Emerging Trends, Sections 1h–1i My first step is to integrate an existing benchmark into HAL and learn the evaluation workflow. After that, I’d like to build a memory benchmark of my own. I haven’t settled on its scope yet, but I want it to help explain how remembered information affects an agent’s decisions on behalf of a user.

If you’re interested in this research, please reach out. I’d love to exchange ideas and collaborate on agent evaluation and memory.