An internal support assistant correctly answers almost every question in a test set. Then an employee asks it to summarize a document that belongs to another team. The assistant provides a confident summary, complete with references. The answer is accurate, and the application has still failed in a way that should stop the release. A single overall quality score would make that failure easy to overlook.
AI safety and evaluation tools help teams turn failures like this into repeatable tests and decisions. Their value depends on which part of the application they can exercise, what evidence they preserve, and how clearly they show the difference between a rough answer and a broken control. A team evaluating a writing assistant needs different release evidence from a team giving an agent permission to update customer accounts.
Start with the decision the test will inform
A useful safety program identifies what can go wrong, who could be affected, and who has authority to accept the remaining risk. The NIST Generative AI Profile provides a voluntary framework for organizing that work across the system lifecycle. Translate broad risk categories into situations the application can actually encounter. An employee sees a restricted benefit policy. A public assistant presents a fabricated refund exception. A model interprets text inside a retrieved attachment as an instruction to call an external tool. Each case needs a different test and a different response. NIST Generative AI Profile
Write each important case with a starting state, the user’s role, the input, the expected result, and the evidence a reviewer should see. For the restricted document case, the expected result might specify that the document never enters the model context and its title never appears in the answer. Run the same request as an authorized user to check that the restriction works without suppressing legitimate access. An application that refuses every request could pass a leakage-only test while being useless to its intended audience.
Separate checks that block release from checks that guide improvement. A confirmed unauthorized disclosure may be a release blocker. A small increase in unnecessary clarifying questions may be acceptable while the team investigates. Record the severity and scope of a failure alongside its frequency. Averaging the two outcomes into a single score discards the distinction the release owner needs.
Match the test method to the failure
| Evaluation method | Evidence it produces | Example |
|---|---|---|
| Deterministic assertion | A repeatable pass or fail for an observable property | A read-only identity never reaches a write endpoint |
| Human review with a rubric | A contextual judgment and a reason for it | An answer addresses a distressed user appropriately |
| Model-assisted grading | A scalable estimate of a defined quality criterion | A response appears to support each claim with supplied evidence |
| Adversarial testing | Examples of inputs and paths that defeat an assumed safeguard | Instructions embedded in a document influence a privileged action |
| Production monitoring | Signals from the deployed system and its real operating conditions | A rise in escalations after a change to the document index |
Use ordinary programmatic checks wherever you can. A forbidden tool call, an invalid field, or a citation pointing to a missing document can often be detected directly. Save human attention for contextual judgment, such as whether a response responsibly communicated uncertainty. When using a model grader, calibrate it against human decisions on a varied sample. Include borderline cases and obvious failures. Give reviewers a way to flag disagreements instead of silently treating the grader’s output as truth.
The boundary under test matters as much as the score. A prompt benchmark tells you how a model responds to a particular input. An application test can include document retrieval, identity, redaction, model routing, and tool execution. If the failure depends on a user’s role or the state of an account, sending the prompt straight to a model skips the components most likely to cause it.
Where Promptfoo and Inspect AI fit
Promptfoo is an example of a configurable framework for evaluating and red-teaming AI applications. Its red-team configuration gives teams a way to select attack plugins and strategies and point the tests at a target. It can suit an application team that wants to turn an adversarial discovery into a recurring regression check. Inspect where attack generation and grading run, which providers receive data, and which parts of the application the target actually exercises. The project’s red-team documentation describes configuration choices that can affect this data flow. Promptfoo red-team configuration
Inspect AI provides building blocks for defining evaluation tasks, running models through them, and scoring the resulting behavior. It is a useful candidate when a team wants to express evaluations in code and control the task, execution, and scoring logic. A team comparing model behavior across different conditions may value that separation. The application owner still has to represent the real deployment in the evaluation when a result depends on permissions, external tools, or application state. Inspect AI documentation
A shared evaluation platform may serve a different need. Teams with several products may spend more time assigning reviews, comparing runs, and making results available to decision makers than writing test cases. Look at whether developers can reproduce a failure from the platform’s records and whether reviewers can open a case without seeing data they are unauthorized to access. A dashboard is valuable when it shortens the path from a failed case to an accountable decision.
Build an evaluation set that ages well
Maintain a stable set of common cases so results remain comparable. Add cases drawn from incidents, support escalations, and near misses. Include examples where the application should answer, refuse, ask for missing information, and hand the task to a person. Include legitimate requests that resemble attacks, such as a security analyst asking the assistant to explain a suspicious email. These cases help detect controls that block the work the system was built to support.
Keep a separate set for exploratory testing. If developers tune prompts or thresholds against every example they will later use to approve the release, improvements may reflect familiarity with those cases. Rotate representative held-out cases into release review and record when you change the set. Track the model configuration, prompt, dataset revision, and any dependencies that influence results. For a retrieval application, preserve enough information to identify which document revisions were available during the run.
Some failures appear intermittently. Repeat high-consequence scenarios where the outcome can vary, then report how often the problem appeared under the tested conditions and how it was reproduced. A small clean sample establishes what happened in that sample. If the consequence is severe, use the result to investigate the control directly and exercise it under additional conditions. A permission check in the tool service is easier to verify than a promise in a prompt.
Treat evaluation records as sensitive data
Prompts, traces, and grader requests can contain customer data, internal documents, or credentials entered by mistake. Follow one sample through the system before adopting a tool. Check what the test runner stores, what the model or grading provider receives, what reviewers can see, and what remains in exported results. Synthetic fixtures are useful for exercising known boundaries. Redacted incident examples can preserve a failure’s essential structure when realistic language matters.
Estimate operating costs using a representative release run. Model calls, attack generation, repeated sampling, and human review each consume resources. A fast suite of deterministic checks can run often, with larger adversarial and human-reviewed suites reserved for changes that alter the model, access boundary, or use case. Assign an owner to investigate failing runs and another person with authority to make the release decision where the risk warrants it.
For a pilot, choose one workflow where the application can cause a meaningful adverse outcome. Introduce a controlled defect in a test environment and run the evaluation against it. Then restore the control and run the authorized counterpart. The tooling earns its place when it catches the defect, preserves enough evidence for someone else to understand it, and gives the release owner a clear decision.