For years, software QA teams relied on deterministic assertion libraries. Inputs were known, state transitions were predictable, and a failing test point led straight to a single line of code. When teams started evaluating single-prompt Large Language Models, unit testing shifted toward assertion functions, measuring faithfulness, hallucination rates, and semantic relevancy across isolated inputs and outputs.
However, as enterprise engineering shifts from simple Q&A models to multi-step autonomous workflows, those traditional testing patterns are breaking down. In autonomous multi-turn systems, the hardest failure modes do not occur in isolated prompts. They emerge across long tool chains, memory retrievals, dynamic API calls, and sandboxed execution environments.
This reality became starkly visible this month. At Black Hat 2026, OpenAI disclosed that internal evaluation agents discovered and exploited a misconfigured Artifactory repository during testing, subsequently coordinating across short-lived sandboxes to access third-party services. Shortly thereafter, news surfaced that development on their multi-agent system, Astra, was paused after internal evaluations indicated potential for autonomous exploit generation and cyber-risk behaviours without human oversight.
When evaluation agents can chain minor environment privileges into unauthorised access, test sandboxes and credential stores can no longer be treated as harmless experimental playgrounds. They are part of the attack surface.
The Breakdown of Single-Turn Evals in Production
Most engineering teams starting out with agent evaluation rely on frameworks like Promptfoo or DeepEval. While Promptfoo excels at YAML-driven configuration and red-teaming security scans, and DeepEval brings a pytest-native approach to RAG metrics, evaluating an autonomous agent requires looking beyond single-turn input and output pairs.
In a recent production case study on AI Evaluation Engineering, an enterprise agentic system passed every automated evaluation metric during offline testing. Faithfulness scores were near perfect because every response was logically grounded in retrieved context. Yet in live operations, the system routinely failed users. The root cause was incomplete context retrieval: the agent constructed plausible-sounding, grounded answers from partial data, masking a critical upstream retrieval flaw.
This disconnect highlights why standard benchmarks are losing predictive value for real-world reliability. According to research on AI Benchmarks in 2026, conventional foundation model benchmarks suffer from severe saturation and benchmark gaming. More importantly, enterprise agentic systems show up to a 37% gap between static lab benchmark scores and real-world deployment performance.
Shifting from Output Metrics to Trajectory Evaluation
To build reliable agentic systems, QA engineers must shift from evaluating final text outputs to auditing the agent’s execution trajectory. Trajectory evaluation measures four critical dimensions across multi-turn interactions:
Step Efficiency: Does the agent solve the task in optimal steps, or does it enter infinite loops and redundant tool calls?
Context Drift: Does the agent maintain its original system prompt and objective across five, ten, or twenty conversation turns without hallucinating or losing context?
Tool Parameter Sequence Integrity: Does the agent select valid tool schemas, pass correctly typed parameters, and handle API errors gracefully?
Egress and Boundary Enforcement: Is the agent constrained from escalating its permissions, reading unauthorised network endpoints, or executing destructive side effects?
Practical Guardrails for Quality Engineers
If you are leading QA or reliability engineering for autonomous agents, here is a practical playbook to implement this week:
Treat Every Tool-Using Agent as an Untrusted Actor: Restrict database credentials, scope API permissions strictly to the least privilege required, and enforce egress firewalls on sandbox test environments.
Build Multi-Turn Adversarial Test Suites: Do not rely solely on single-turn jailbreak prompts. Design test cases that simulate multi-step social engineering, context poisoning, and tool parameter manipulation.
Track Trajectory Length Distributions: Look at long-tail latency and step distributions rather than mean metrics, as average scores hide catastrophic multi-turn loops.
Implement Circuit Breakers: Define explicit kill-switches and step-count limits so an agent that loses its reasoning loop is safely halted before consuming tokens or invoking runaway API calls.
I’m Ajas Bakran. A decade in software testing and quality engineering, now focused entirely on AI agent evaluation and reliability. I write about what’s actually working in this space, and what isn’t, every week here.