Key takeaways
- Evaluate the completed state, tool trajectory, policy compliance, side effects, recovery, latency, and cost—not the final sentence alone.
- Build cases from real exceptions and operational risk, then keep a stable regression set beside fresh exploratory tests.
- Measure task utility and security together; a safe refusal that prevents all useful work is not a production success.
- Use automated graders where they are reliable, but calibrate them against human review and deterministic checks.
01
Start with the work people actually hand over
Collect representative requests from support, marketing, sales, or operations. Include clean examples, missing fields, contradictory records, unclear ownership, stale data, policy exceptions, repeated requests, and requests the system must refuse or escalate. Strip or synthesize sensitive details before the cases enter a reusable test set.
Define the acceptable end state before running a model: which records may change, which facts must be preserved, what must be shown to a person, and what must never happen. This avoids grading a polished explanation while an incorrect tool call sits unnoticed underneath it.
02
Score the trajectory as well as the destination
AgentDojo evaluates realistic tasks and prompt-injection attacks together, and its results show why utility and security cannot be collapsed into one pleasant-looking score. A run can reach the right business outcome through an unsafe path, or refuse so often that no useful task survives.
Capture tool selection, arguments, authorization decisions, retries, intermediate state changes, approvals, final state, and recovery. Use deterministic assertions for exact requirements such as “no email was sent” or “only this deal field changed.” Reserve human or model judgment for qualities that genuinely need interpretation.
Swipe to compare every column
| Measure | Question | Useful evidence |
|---|---|---|
| Task success | Did the requested state become true? | Authoritative system state |
| Policy compliance | Were boundaries and approvals respected? | Policy decisions and tool log |
| Path quality | Was the route efficient and defensible? | Trajectory, retries, latency, cost |
| Recovery | Did failure leave a safe, usable state? | Compensation and incident record |
03
Make the suite hard to game
NIST has documented contamination and grader gaming as evaluation risks. Keep hidden variants, rotate surface details, inspect sampled traces, and check that success rules match the intent of the task. Do not let the system read answer keys, grader code, or filenames that reveal the expected action.
Run tests against the exact model, instructions, tools, policies, and retrieval configuration intended for release. A model-only benchmark does not validate the application that surrounds it.
04
Treat production failures as new tests
Every reviewed incident or near miss should become a sanitized regression case. Track results by case family and severity rather than hiding a dangerous failure inside one average. Compare releases, investigate regressions, and require sign-off when a change expands authority or introduces a new data source.
The goal is not a frozen benchmark that always turns green. It is a living body of evidence that makes the next release less surprising.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
- AgentDojo: Dynamic Evaluation of Prompt Injection Attacks and Defenses
- NIST: Strengthening AI Agent Hijacking Evaluations
- NIST: Cheating on AI Agent Evaluations
- NIST AI Risk Management Framework
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
