AI, Honestly
AI Demos Always Work. Here Is How to Test Before You Buy.
Every AI demo works. I have sat through dozens of them, as a fractional CTO and AI advisor, and I have never once seen one fail on stage. That is not because the technology is flawless. It is because the vendor chose the example.
If you are an owner or operator trying to decide where AI fits in your business, that gap between the demo and your Monday morning is the most important thing to understand. It is also the cheapest thing to test, if you know how.
Why demos always work
A demo is a rehearsed path. The vendor picks the cleanest input, the friendliest scenario and the question the system answers best. Nothing in the room is messy, because mess was removed on purpose.
Your business does not run on the rehearsed path. It runs on the customer who sends a photo of a handwritten note, the order that fits no category, the invoice with two reference numbers, and the client who changes their mind on Thursday. Those cases are where time and money actually go, and they are exactly what a demo avoids.
Change the demo before you buy
You do not need a technical background to run a better test. You need ten real cases from last month.
- Pull ten real examples of the work you want to hand over.
- Make sure at least three of them are the ugly ones: missing information, odd requests, unhappy customers.
- Ask the vendor to run those cases live, while you watch, with no cleanup first.
How a tool handles the ugly three tells you more than the polished pitch ever will. If it falls over, that is a useful answer, and it just saved you a year.
The pilot trap
A pilot is a longer demo, and it has the same flaw. It usually runs with your most enthusiastic person, on hand-picked cases, with someone quietly fixing every miss. Of course it works.
Then you roll it out to the whole team on every case, and nobody is fixing anything anymore. The misses that were invisible in week two are the ones your customers find in week ten.
A pilot only counts if it passes three tests:
- Run it on a random week of real work, not your best cases.
- Put your biggest skeptic in charge of checking it, not your biggest fan.
- Stop fixing the outputs by hand, and count what actually goes wrong.
If it still works after that, you have something real. If it does not, you found out for the price of a pilot instead of the price of a rollout.
Shrink the question
When AI fails, the instinct is to reach for something bigger. Experience says the opposite. On a sports video project, we used face recognition to find the marketable moments in a match. Pointed at every Premier League player, it produced garbage, and at one point it tagged a player who had been dead for twenty years.
The fix was not more AI. It was a constraint. In any match there are only 22 players on the pitch, so we told the system to choose from those 22 and nothing else. Accuracy jumped almost overnight.
Most small business AI failures work the same way. The tool is asked to be right about everything, so it is confidently wrong about something. Narrow the job to a short list and it gets sharp.
What to automate first
The safest first jobs are the ones with the fewest exceptions: booking, reminders, receipts, the same ten questions asked all day. Jobs where the honest answer is "it depends" should stay with a person, at least until the AI has earned your trust on the simple work.
Before you automate anything, count how often that job has an exception. That single number predicts how the project will go better than any feature list.
Where does AI fit in your business?
After 35+ years building software and security systems, and running technology for three companies of my own, I use this lens to separate AI hype from what pays off. I take on a few clients at a time as an AI advisor and fractional CTO, and I spend half that time talking people out of AI they do not need.
Book your free 15-minute AI Fit Call and we will look at where AI genuinely fits in your business, and where it does not.
Not ready for a call? Start with the free AI Fit Audit.
Stephen Bickle · Nov 15, 2026