How to Assess an AI Agent Vendor: Security, Data and Governance Questions
The questions to ask before buying an AI agent platform or product: data use, security, identity, permissions, controls, evaluation, audit, incidents and exit.
Quick answer
Assess an AI agent vendor on five areas: data and privacy (training use, residency, retention, subprocessors), security and identity (how agents authenticate, delegated access, least privilege), controls and audit (approvals, limits, action logs you can export), quality and operations (evaluation, model change management, monitoring, incident response) and commercial and exit (pricing at your volume, data and configuration export, open standards). Ask for evidence, not just answers, and pilot the product in a sandbox with your own adversarial and failure scenarios before granting production access.
Why agent vendors need extra scrutiny
A standard SaaS security review asks whether a vendor protects your data. An agent vendor also acts inside your systems: it may read mailboxes, update CRM records, issue refunds or send messages on your behalf. That raises questions ordinary questionnaires do not cover: whose identity it acts under, what stops it doing too much, whether a manipulated document can redirect it, and whether you can prove afterwards what it did. The OWASP Top 10 for Agentic Applications is a useful lens for these risks; see the OWASP agentic guide.
The assessment questions
| Area | Questions to ask | Evidence to request |
|---|---|---|
| Data use | Is our data used to train or improve models? Can we opt out? How long are prompts, outputs and logs retained? | Contract terms, data processing agreement |
| Data location | Where is data processed and stored? Which subprocessors and model providers are involved? | Subprocessor list, region options |
| Identity | How does the agent authenticate to our systems? Does it act on behalf of users with scoped, revocable delegation? | Architecture docs, OAuth scopes |
| Permissions | Can we restrict tools and actions per agent and per user? Are limits enforced in code? | Admin console demo, configuration docs |
| Approvals and limits | Can consequential actions require approval? Are there spend, volume and rate limits? | Demo of approval flows and limits |
| Audit | Is every action logged with identity, inputs, outputs and approver? Can we export logs to our systems? | Sample audit export |
| Manipulation resistance | How is prompt injection from documents, emails and web content handled? | Security documentation, test results |
| Quality and change | How are model and prompt changes evaluated and communicated? Can we pin versions? | Release notes process, evaluation approach |
| Incidents | How are incidents detected, contained and notified? Is there a kill switch we control? | Incident policy, SLAs |
| Certifications | Which assurance reports and certifications cover the product? | SOC 2 report, ISO/IEC 27001, ISO/IEC 42001 certificates |
| Exit | Can we export data, configurations and logs? What happens to our data on termination? | Contract exit clauses |
Key takeaway
Certificates show that a vendor runs a management system. Only a demonstration and a pilot show whether the agent's permissions, approvals and logs work the way you need.
Run a structured pilot
Before production access, pilot the vendor in a sandbox or test tenant with masked data: normal cases from your own workflows, edge cases, tool failures and adversarial inputs such as instructions hidden in documents. Confirm that restricted actions are refused, approvals trigger, limits hold and audit logs capture what happened. See AI agent sandbox for how to set up the environment.
Scoring and decision
Weight areas by your risk tier. For an agent that only drafts internal summaries, data use and retention dominate. For an agent that acts on customer accounts or money, identity, permissions, approvals and audit become pass/fail requirements. Record the assessment, the evidence and any accepted risks with an owner, and schedule a re-assessment, since vendors change models and features frequently. If gaps are fundamental, reconsider whether to integrate, customize or build instead; see build vs buy AI agents.
Evaluating AI agent platforms?
ZSpace Labs helps teams assess AI agent vendors against their own workflows, runs sandbox pilots and builds the integrations and controls around the chosen platform. See AI automation services.
Common mistakes
- Using a generic SaaS questionnaire with no agent-specific questions
- Accepting a certificate as proof that agent actions are controlled
- Granting broad OAuth scopes during setup "to make it work"
- No pilot with adversarial or failure scenarios
- No exit plan for data, configurations and logs
Red flags
- Vague or changing answers about whether your data trains models
- Agents that require broad administrator access to work
- No way to require approval for consequential actions
- Action logs that cannot be exported or lack user and input detail
- No documented approach to prompt injection
- Model changes rolled out without notice or a way to pin versions
- No data export or deletion commitments at contract end
Contract points to request
Ask legal and procurement to cover the agent-specific points, not just standard SaaS terms: no training on your data without explicit opt-in; data residency and retention limits; subprocessor change notice; security incident notification timelines; notice of material model or behaviour changes; audit log retention and export; service levels for the agent's availability and support; liability and indemnities appropriate to the actions the agent can take; and exit assistance with export of data, configurations and logs. For who carries responsibility toward your customers regardless of contract terms, see who is responsible when an AI agent makes a mistake.
Conclusion
Buying an AI agent means letting a vendor's software act inside your business. Assess data use, identity, permissions, approvals, audit, quality management and exit, ask for evidence, and prove the controls in a pilot. For the governance standard many vendors cite, see ISO/IEC 42001, and for identity requirements, AI agent authentication.
Common questions.
Because agents act. Beyond the usual security and privacy checks, you need to know how the agent authenticates, what it can do in your systems, how its actions are limited, approved and logged, how it resists manipulation and how quality is maintained when models change.