AI vendor evaluation is harder than ordinary software evaluation for one structural reason: the product is probabilistic, and probabilistic systems demonstrate beautifully. A curated demo on clean examples tells you almost nothing about how the system behaves on your Tuesday-afternoon reality.
The questions below are the ones that produce informative answers. What matters is less the specific response than whether the vendor has clearly thought about it before you asked.
Questions about accuracy
How do you measure accuracy, and on what test set?
The answer you want describes a fixed set of labelled examples, a defined scoring method, and results tracked over time. The answer that should worry you is a percentage with no methodology attached, or a shrug toward customer satisfaction. A vendor without an evaluation harness cannot tell you their quality, because they do not know it.
What does the system do when it is uncertain?
Well-built systems have a confidence mechanism and an escalation path. Poorly built ones return their best guess with the same authority regardless. Ask to see what an escalation looks like in the interface.
Show me your worst cases
This is the most informative question on the list. A vendor with a real system in production has a catalogue of failure modes and will describe them readily, because they have spent months working on them. A vendor without one will deflect. The willingness to answer matters more than the answer.
Questions about your data
- Is our data used to train or fine-tune any model, yours or a third party's? Get this in the contract, not in an email.
- Which subprocessors touch our data, and in what jurisdictions is it processed and stored?
- What is retained after processing, for how long, and can we require deletion?
- If we terminate, what happens to our data and any model artefacts derived from it?
- Is there an audit log of what the system did, sufficient to reconstruct a specific decision months later?
The audit question is the one most often overlooked and most often needed. When a customer disputes an outcome, or a regulator asks, or an internal investigation starts, you need to be able to say what the system saw and what it produced on a specific date. Systems that cannot do this create a category of problem that has nothing to do with accuracy.
Questions about change
Underlying models change. They are upgraded, retired, and re-tuned by their providers on schedules the vendor does not control. This is the most underexamined risk in the category.
| Ask | A good answer sounds like | A concerning answer sounds like |
|---|---|---|
| What happens when the underlying model is upgraded? | We re-run our evaluation suite and hold the previous version until the new one passes | We get the improvements automatically |
| Can we pin a version? | Yes, with a defined support window and notice before deprecation | You are always on the latest |
| How will we know if quality drifts? | Scheduled scored runs, alerting on a threshold, visible to you | Customers tell us |
| Who owns prompts and configuration built for us? | You do, and we will export them | That is our proprietary layer |
The last row deserves attention. If the accumulated configuration, prompts, and rules that make the system work on your business are the vendor's property, then your switching cost grows every month you use it and you have no leverage at renewal.
Questions about cost
- How is this priced: per seat, per run, per token, or flat? Model it at three times current volume.
- What happens if volume spikes unexpectedly? Is there a ceiling, and who is liable above it?
- Are there charges for reprocessing when something fails and has to run again?
- What is included in support, and what is billed?
Usage-based pricing is fair in principle and occasionally alarming in practice, because a workflow that suddenly runs ten times as often produces a bill nobody approved. Ask for a hard ceiling with alerting, and treat reluctance as information.
Questions about the exit
Price the exit before you sign. Not because you expect to leave, but because a relationship you cannot leave is a relationship you cannot negotiate.
- Can we export our data in a documented, usable format, and is there a charge?
- Do we keep the prompts, rules, and configuration developed for us?
- What is the notice period, and what happens operationally during it?
- If you are acquired or shut down, what are our rights, and is there an escrow arrangement?
If you cannot describe how you would leave a vendor, you are not a customer with options. Your renewal conversation will reflect that.
Signals that should slow you down
- Accuracy quoted as a single number with no test set described.
- A refusal to run a paid pilot on your own data.
- No named failure modes, or visible discomfort at the question.
- Quality monitoring described as something the customer does.
- Contract language that is silent on training use of your data.
- Configuration built for you that is claimed as the vendor's property.
- Pricing that has no ceiling and no alerting.
None of these are automatically disqualifying. Early companies genuinely do have thin evaluation practice, and a strong team with a weak harness can be a good bet. But each one should move the price, the pilot conditions, or the contract terms in your favour, and a vendor who bristles at the questions is showing you how the relationship will run.
The shortest version
If you do only three things: insist on a paid pilot against your own data with a pre-agreed target, get data use and model-change behaviour in writing, and make sure the configuration built for your business belongs to you. Those three cover most of the ways these purchases go wrong.