Your Marketing Agent Passed the Demo. Now Give It an Eval.

A demonstration usually begins with clean inputs, available tools, and a cooperative user. Marketing operations rarely does. The CRM contains conflicting records, the offer changed yesterday, a source is stale, an ad platform rejects a claim, and the approver is away. An agent should be evaluated in the environment where those conditions occur.

OpenAI’s 2026 evaluation guidance emphasizes that performance depends on the harness and workflow around the model. Define the system under test: model and version, instructions, available tools, permissions, retrieval sources, state, timeouts, approvals, and fallback. A model score without that context is not a deployment claim.

Collect representative tasks from campaign briefs, routing exceptions, rejected claims, customer corrections, tool errors, and successful work. Include ordinary volume as well as high-impact edge cases. Remove or protect personal data, preserve the minimum context needed to judge the task, and record the expected behavior rather than one exact sentence.

Separate capability from policy. “Can the agent identify a pricing contradiction?” is different from “May it change the published price?” The first can be scored for detection; the second should be scored for escalation and non-action. This distinction keeps a capable model from being rewarded for exceeding its authority.

Use exact or programmatic checks for schema, required fields, source presence, numerical calculations, allowed tools, state changes, and prohibited actions. Use a rubric for tone, usefulness, strategic reasoning, or whether a summary preserves the important condition. Calibrate model-based graders against human reviewers instead of assuming fluent judgment is consistent judgment.

Score the trajectory as well as the final response. An agent may produce a correct report after querying an unauthorized source, retrying a chargeable tool twenty times, or overwriting a newer record. The end result hides the operational failure unless tool calls, latency, cost, approvals, and state changes are part of the evaluation.

Run a stable regression set before a model, prompt, tool, policy, or knowledge change reaches production. Add a smaller exploratory set for new failure modes. Compare results by task family and severity; an unchanged average can hide a serious regression in consent handling or record updates.

Every production incident should create three artifacts: a corrected customer or business state, a root-cause note, and a test that would catch the failure next time. Expansion of authority should follow evidence on representative work, not confidence from the latest demo. The eval is not a launch gate that disappears after launch; it is the memory of the system.