Evaluate an Agent Vendor Before Connecting Production Data

The decision this guide should improve

A useful guide changes a real decision. For this topic, the decision is whether and how to compare security, evaluation, observability, portability, and exit controls before a purchase. Name who owns that call, the options they can choose, the deadline, and what would make them change course.

Keep observations separate from assumptions. An official source can explain a standard or risk, but it cannot prove what happened in your particular product, portfolio, job, home, or mini-app. Direct evidence needs its own date, subject, method, and limitation.

Use these current primary sources

The workflow below is grounded in NIST AI Risk Management Framework, NIST Generative AI Profile, OWASP Excessive Agency, OWASP Top 10 for Agentic Applications. Open the source that supports the exact claim you need, confirm its scope and update date, and record where it does not apply. A smaller claim-to-source map is more useful than a decorative bibliography.

Five-step workflow

1. 1. Write the intended use and prohibited data. Write the evidence, owner, date, and next decision beside the step.
2. 2. Request evidence for security and model claims. Write the evidence, owner, date, and next decision beside the step.
3. 3. Test permissions and isolation in a sandbox. Write the evidence, owner, date, and next decision beside the step.
4. 4. Verify logs, export, deletion, and incident terms. Write the evidence, owner, date, and next decision beside the step.
5. 5. Run an exit rehearsal before dependency grows. Write the evidence, owner, date, and next decision beside the step.

Do the steps in order the first time. If a safety, privacy, financial, health, legal, or authorization boundary appears, pause the workflow. More activity is not progress when the prerequisite is missing.

Worked example

A team tests two agent platforms with synthetic customer data. One has stronger demos but cannot export traces or restrict tool scopes. The scorecard records the gap before contracts and integrations make switching expensive.

The useful pattern is the visible chain from context to evidence to decision. Another person should be able to understand what was observed, what remained uncertain, and why the next action was proportionate.

Decision record

Common failure patterns

• Starting with a tool. Start with the user outcome and choose the lightest tool that produces credible evidence.
• Treating exposure as destiny. A risk, score, or capability describes a condition; it does not predict every outcome.
• Moving the threshold afterward. Set success and stop conditions before seeing the result.
• Hiding exceptions. Preserve manual corrections, failures, disagreements, and missing data.
• Reporting activity as impact. A click, test run, generated answer, or submitted form is not automatically the user outcome.

Questions for a second reviewer

1. Which statement is most likely to be wrong?
2. Which evidence is direct, and which is inferred?
3. Does every requested permission or data field support the stated job?
4. Who bears the cost of a false positive or false negative?
5. What would cause us to stop proudly?
6. Can someone reproduce the result from this record?

Sources and review note

• NIST AI Risk Management Framework
• NIST Generative AI Profile
• OWASP Excessive Agency
• OWASP Top 10 for Agentic Applications

Source links reviewed 2026-07-13. Follow each publisher for revisions and confirm that the guidance applies to your location and situation.

Related guides

• Build an AI Agent Evaluation Plan
• The Agent Tool-Permission Matrix
• AI Agent Red-Team Runbook