Six questions I would ask any decision intelligence platform
By Meet Patel · 2026-09-06 · updated 2026-09-09
Summary
Six evaluation questions for decision intelligence platforms: (1) what did it decide not to tell me and on what threshold, (2) what happens when two sources disagree, (3) what happens when nobody responds, (4) can it say it does not know, (5) does it check whether the decision worked, (6) who is accountable when it is wrong. Five of the six are organisational rather than technical questions.
Key Metrics & Takeaways
- 6 questions
- the evaluation checklist
- 5 of 6
- are organisational, not technical
- 3 sources
- typical number that must be reconciled before an answer is trustworthy
I build a product in this category, so everything below is biased. Read it with that in mind and use it anyway, because the questions are ones I would rather be asked than not.
Most evaluations of this software go wrong the same way. The buyer runs a feature comparison — connectors, query speed, chart types, model support — and every vendor scores well, because those are the parts that are easy to build and easy to demo. Then the tool goes in, and eighteen months later it is another surface nobody opens.
These six questions are the ones that actually separate products. Five of them are not technology questions, which is the most useful thing I can tell you about the category.
1. What did you decide not to tell me, and on what threshold?
This is the whole thing in one question.
Ask the vendor to show you what the system suppressed last week and why. If the product is real, selection is its core function and there is a record: four hundred things observed, six raised, and a stated basis for the other three hundred and ninety-four.
If the answer is a shrug, or a vague reference to relevance scoring, you are buying a reporting tool. There is nothing wrong with a reporting tool. Just do not pay decision-layer prices for one.
Follow up by asking who can change the threshold. If it is the vendor, your company's judgment is being outsourced to a product roadmap.
2. What happens when two of my systems disagree?
Bring a real metric. Something your CRM, your billing system and your finance model each compute differently — most companies have several, and net revenue retention is almost always one of them.
There are three possible responses and only one of them is good.
A product might show you all three numbers, which is honest and useless: you have moved the argument into a nicer interface. It might show one number with no explanation, which is worse, because it has silently picked a definition and you will find out which one during a board meeting. Or it names the definition it used, shows where the others differ and why, and tells you which question each is the right answer to.
Only the third has done any work. The other two have relocated your problem.
3. What happens when nobody responds?
Silence is the normal case, not the edge case. People are busy, things get read on a phone and forgotten, owners go on leave.
So ask what the system does on day three of no acknowledgement. Does it escalate to someone else, and to whom, and did anyone in your company agree to that? Does it quietly expire? Does it repeat, which trains everyone to filter the channel within a month?
This is the question vendors are least prepared for, and it predicts adoption better than anything on the feature matrix. A system that does nothing about silence will be silent about everything within a quarter.
4. Can it tell me it does not know?
Ask for a case where the data is genuinely insufficient — a segment with eleven customers, a channel with two weeks of history — and watch what comes back.
A system that always produces a confident answer is producing confidence. That is a different product, and in an operating context it is a dangerous one, because the whole value of the thing is that you stop personally verifying every claim.
I want to see a stated confidence, the sample it rests on, and an explicit "not enough to act on". Products remove this because it demos badly. It is the single feature I would trade the most for.
5. Does it check whether the decision worked?
A recommendation with no follow-up is untested advice delivered at volume.
Ask to see the record: what was recommended, whether anything was done, what happened afterwards, and whether the system's hit rate is visible to you. Not to the vendor — to you.
Almost nothing does this, and I understand why. Publishing your own hit rate is an unpleasant thing to build. It is also the only mechanism by which a system like this gets better rather than louder.
6. Who is accountable when it is wrong?
It will miss something that mattered. Assume that, and settle the question before you buy rather than in the incident review.
Was it a vendor failure — the system had the data and did not connect it? A threshold failure — someone in your company set the bar so the thing was correctly ignored? Or an ownership failure — it was raised, to a real person, who did nothing?
Those have completely different fixes, and if nobody has agreed in advance how to tell them apart, the answer defaults to blaming the tool. Which is comfortable, and which guarantees the underlying problem stays.
The part that surprised me
When I first wrote a version of this list for our own use, I expected it to be a technical evaluation and it kept refusing to be one.
Five of the six are questions about your company. Who sets thresholds. Which definition is canonical. What escalation is permitted. Whether anyone reviews outcomes. Whether being wrong has an owner.
A company that has answered those will get value from almost any competent product in the category. A company that has not will get value from none of them, and will conclude the category does not work. I have watched that happen and initially blamed the software, which was the easy read and the wrong one.
Steps
- Ask what it decided not to tell you — Require the vendor to show the things the system suppressed this week and the threshold that suppressed them. A reporting tool has no answer because it never selected anything.
- Ask what happens when two sources disagree — Give it a metric your CRM, billing system and finance model each compute differently. A product that returns one number without naming the definition it used is hiding the disagreement rather than resolving it.
- Ask what happens when nobody responds — Silence is the normal case in a busy company. Find out whether an unacknowledged item escalates, expires, or simply repeats until people learn to ignore the channel.
- Ask whether it can say it does not know — Request a case where the data is insufficient. A system that always produces a confident answer is producing confidence, not evidence.
- Ask whether it checks afterwards — A recommendation with no outcome review is untested advice. Ask to see the record of what was recommended, what was done, and what happened next.
- Ask who is accountable when it is wrong — Establish before purchase whether a missed signal is a vendor failure, a threshold failure or an ownership failure, and who reviews it. If nobody owns being wrong, nobody will improve it.
Frequently asked questions
What should you look for in a decision intelligence platform?
Look for evidence of selection and accountability rather than breadth of features. Ask what the system chose not to surface and on what threshold, how it resolves two sources that disagree, whether it can report that it does not know, and whether it checks after the fact whether a decision worked. A product that cannot answer those is a reporting tool with a new label.
How is decision intelligence software different from BI software?
BI software is judged on whether the answer it returns is correct. Decision intelligence software is judged on whether it raised the right thing, to the right owner, with evidence, in time to matter. The evaluation criteria are therefore different: coverage and query speed matter less than selection quality, reconciliation and follow-through.
Who should own a decision intelligence platform inside a company?
Someone accountable for operating decisions rather than for data — typically a COO, CRO or head of operations, with data engineering supporting. Owned by the data team alone, it becomes another reporting surface, because the thresholds that determine what deserves attention are management judgments and need a management owner.
Written by Meet Patel — founder of Company 8, building Dan (usedan.com). Dubai, UAE.