You do not need a procurement department to evaluate an AI vendor. You need a list of questions and the patience to read the answers properly.
Here is the list. Copy it, send it, and watch for a single pattern: artifacts, or adjectives.
The short answer
If you only send six, send these:
- Will you sign a BAA?
- Who are your subprocessors?
- Is our data used to train models?
- What is your breach notification timeline, in hours?
- Show us a single agent action from six months ago.
- What happens to our data when we leave?
A vendor answering these with documents is worth the longer conversation. One who cannot will not improve with more questions.
The full 21
Contracts and legal (1–4)
- Will you sign a business associate agreement? If no, they cannot touch PHI. Full stop.
- May we see your standard BAA before we commit? Read it against HHS's required provisions.
- Does the BAA bind your subcontractors to the same terms? HHS requires this; confirm it is present rather than assumed.
- What is your liability cap, and does it carve out data breaches? A cap set at one month's fees means the risk sits with you.
Data handling (5–9)
- Which specific fields does the agent receive? Minimum necessary applies to software too.
- Is our PHI used to train or fine-tune any model — yours or a third party's? Get the clause number, not a reassurance.
- Where is data stored and processed, by region?
- Is data encrypted in transit and at rest? Addressable under the Security Rule, expected in practice.
- Is our data isolated from other customers? Ask how, not whether.
Subprocessors (10–12)
- Name every subprocessor that touches PHI. Model provider, cloud host, telephony, document processing.
- Will you notify us before adding or changing one? Inference providers change invisibly.
- Do you have a BAA with each of them?
Logging and audit (13–15)
- Walk us through one agent action from six months ago. Input, logic version, output, reviewer, timestamp. This is the single most revealing question on the list.
- Are prompts containing PHI logged, for how long, and who can read them? The most-overlooked PHI store in AI systems.
- Can we export our audit logs? In a format an auditor accepts.
Security posture (16–18)
- Do you have a SOC 2 report? Which type, what period, what scope, any exceptions? Ask for the report under NDA, not the badge.
- When was your last security risk analysis, and may we see a summary? OCR repeatedly identifies deficient risk analysis as among the most common failings it finds.
- Has an outside party tested your systems recently?
Incidents and exit (19–21)
- What is your breach notification timeline, in hours? "Promptly" is not a commitment.
- Have you had a security incident affecting customer data? What happened? A vendor who describes one clearly is often safer than one claiming a spotless history.
- On termination, is our data returned or destroyed — including backups, prompt logs, and derived data?
How to read the answers
The signal is rarely in whether the answer is impressive. It is in whether it is checkable.
| Pattern | What it means |
|---|---|
| "Bank-grade security" | Describes nothing. No such standard exists |
| "Fully HIPAA compliant" | No certification exists — this is a self-assessment |
| "SOC 2 certified" | SOC 2 produces a report, not a certificate. Ask which type and period |
| "Enterprise-ready" | An adjective |
| "Clause 4.2 prohibits it" | Checkable |
| "Here's the report, under NDA" | Checkable |
| "Here's that action from March, with the version that ran" | Checkable |
One rule covers most of it: any claim that cannot be turned into a document or a demonstration is marketing. That includes ours.
What is reasonable to expect from a small vendor
Do not use this list to disqualify every company that is not a large enterprise. Early-stage vendors often have genuine controls before they have audit reports, and demanding a SOC 2 Type II from a two-year-old company will leave you choosing only from incumbents.
What matters is whether the gaps are stated or hidden. A vendor saying "our SOC 2 audit is underway, here is our risk analysis and our BAA in the meantime" is being straight with you. A vendor with a SOC 2 badge who cannot produce a report is not — and that second case is more common than it should be.
Our own answers
It would be poor form to publish this list and not answer it. As of 7 August 2026:
| # | Question | Our answer |
|---|---|---|
| 1–3 | BAA, shared in advance, binds subcontractors | Yes; available for customers handling PHI |
| 5 | Fields the agent receives | Scoped per agent — eligibility gets coverage and demographics, not clinical notes |
| 6 | PHI used to train models | No |
| 8–9 | Encryption, tenant isolation | Encrypted in transit and at rest; isolated per workspace |
| 10–12 | Subprocessors | Named list on request, with BAAs in place |
| 13 | Reconstruct one action | Yes — skill version, source citation, timestamp, operator on every step |
| 14 | Prompt logs | Retained under policy; sensitive fields redactable in operational views |
| 16 | SOC 2 | In progress. No report issued yet. We will not describe it otherwise |
| 17 | ISO 27001 | Audit underway |
| 19 | Breach notification | Defined in the BAA |
| 21 | Exit | Return or destruction per the BAA |
Two of those rows are "not yet," and we are leaving them that way. While writing this cluster we discovered our own site had described SOC 2 as complete when the audit is not — we corrected it, and the security page now states each framework's real status.
That correction is the reason this post exists in the form it does. The drift from we are working toward it to we have it happens quietly, usually with nobody deciding to lie. Sending this list is how a buyer catches it.
Frequently Asked Questions
What security questions should a small practice ask an AI vendor?
Start with six that filter fast: will you sign a BAA, who are your subprocessors, is our data used to train models, what is your breach notification timeline in hours, can you show us a single agent action from six months ago, and what happens to our data when we leave. A vendor who answers those six with documents rather than adjectives is worth the longer conversation, and one who cannot answer them will not improve under further questioning.
Do I need a procurement team to evaluate an AI vendor?
No. Most of what a formal security review produces can be obtained by sending a written list of questions and reading the answers carefully. The advantage a procurement team has is time and pattern recognition, not access — every artifact that matters is something a vendor will hand to any serious buyer who asks. An afternoon spent on the BAA and the subprocessor list covers most of the real risk.
What is a red flag in a vendor security answer?
Adjectives where an artifact belongs. Bank-grade security, enterprise-ready, fully compliant, and military-grade encryption are all phrases that describe nothing checkable. The specific pattern to watch for is a claimed certification that cannot be produced on request, because a real report is a file a vendor sends under NDA as a routine part of procurement.
Should I ask for a SOC 2 report or is the badge enough?
Ask for the report. A badge does not tell you the type, the observation period, the scope, or whether there were exceptions, and all four determine whether the attestation covers anything you care about. Vendors with genuine Type II reports share them under NDA without friction, so a request that meets resistance has told you something useful.
What should happen to our data when we stop using an AI vendor?
It should be returned or destroyed, and the contract should say which. HHS sample BAA provisions call for return or destruction of PHI at termination where feasible, with protections continuing for anything retained. Ask specifically about backups, prompt logs, and derived data, since those copies are the ones most often missed when a deletion request is processed.
How do I verify a vendor's claims rather than trusting them?
Ask for the artifact behind each claim and check one of them yourself. A SOC 2 claim becomes a report with a date range, an audit-trail claim becomes a walkthrough of one real action from months ago, a no-training claim becomes a clause number, and a subprocessor claim becomes a list. Any claim that cannot be converted into a document or a demonstration should be treated as marketing.
Is it reasonable to ask a small AI vendor for all of this?
Yes, though the answers will differ from what a large vendor gives you and that is not automatically disqualifying. A young company may lack a SOC 2 report while having a real risk analysis, a signed BAA, and clear data handling. What matters is whether the answers are specific and whether the gaps are stated plainly rather than papered over with badges.
What to do next
Send the six. Then send the rest to whoever survives.
If you want the background on any answer you get back, we wrote it up: why "HIPAA certified" does not exist, what a SOC 2 report actually covers, the BAA clauses specific to AI, and where PHI travels during inference.
Send us the list too. Our answers are above, including the two that say "not yet."
