Evaluation
A goal-directed system needs tests that check its behavior and outcomes.
Evaluation is checking an AI system against defined tasks and success criteria. An evaluation may score an answer, inspect actions, or test whether the resulting environment matches the intended outcome.
[Anthropic]In practice
A team gives a booking assistant ten test requests and checks whether each resulting reservation follows the requested dates and constraints.
[Anthropic]A little deeper
A task specifies the goal; a trial is one attempt; a grader scores a result. Repeated trials help reveal variability that a single successful demonstration can hide. [Anthropic]
A common mix-up
One successful demo proves the agent is dependable.
Success on one attempt does not establish performance across tasks or repeated trials. [Anthropic]
Helpful to know: Agent