5 Red Flags Every Business Must Spot Before Buying AI Software

Every week, businesses sign contracts with AI software vendors they will regret.

Not because AI doesn't work. Not because the problem wasn't real. But because the warning signs were visible before they signed — and nobody knew what to look for.

Capterra's 2026 Software Buying Trends Report, based on survey findings of 3,385 software decision-makers across 11 countries, found that only 34% of buyers are successful software adopters. The remaining 66% experience unexpected disruption, regret, or both after a purchase — and nearly nine in ten who regret a software purchase first encountered an unexpected disruption during implementation.

AI software makes this problem worse. The gap between what AI vendors promise in a sales pitch and what actually ships in production is wider than almost any other category of business software. The demos are polished, the case studies are cherry-picked, the language is deliberately vague, and the contracts are signed before anyone has tested the product with real data, real users, or a real workflow.

After evaluating dozens of vendors across consulting engagements, the same warning signs come up again and again. This article gives you the five that matter most — the ones that consistently predict regret — so you can spot them before you sign, not after.

---

Why AI software is harder to evaluate than regular software

Traditional business software has a long track record. You can ask for references, look at Trustpilot and G2 reviews, request a trial, and see the exact feature set before buying. The gap between demo and delivery is well understood.

AI software breaks most of these evaluation tools. A demo can show the model performing perfectly on hand-picked inputs. A case study can highlight one success while quietly omitting ten failures. The feature set includes "AI-powered" claims that are impossible to verify without deep technical access. And the performance on your data — which is almost always different from the demo data — is unknown until after you've paid and started implementation.

Black-box AI that just gives answers without explanation is risky. Better systems show confidence scores or explain their reasoning. This matters for trust and debugging.

The five red flags in this guide are specifically calibrated for AI software evaluation. They go beyond the generic "check the contract" advice and address the AI-specific ways vendors mislead buyers — intentionally or not.

---

Red Flag 1: The demo uses their data, not yours

This is the most common and most consequential red flag in AI software sales, and it is deliberately engineered.

Every AI vendor runs their demos on curated data — clean, representative, pre-processed inputs that the model performs well on. They have been optimising for this demo data for months. The sales team knows exactly which inputs to use and which to avoid. The demo is not a representation of how the software performs on real-world business data. It is a performance.

This is the most common red flag and the hardest to spot. The ask that reveals it is simple: request a live test with your own data before you sign.

Not a slide. Not a pre-recorded video. A live session where you supply the inputs and observe the outputs in real time on data that comes from your actual business — your support tickets, your invoices, your contracts, your customer records. If the vendor declines, delays, or creates friction around this request, you have your answer.

The real-world consequence of skipping this test is severe. A client bought an AI-powered lead scoring tool that had no native integration with their CRM. The vendor said they had a "robust API." What they actually had was a basic REST endpoint with limited documentation and no pre-built connectors. The client ended up spending almost as much on custom integration work as they spent on the tool itself.

What to do: Before the demo, prepare 20 to 30 real examples from your business — a mix of typical cases and edge cases that represent your actual workflow. Require the vendor to run the tool on your inputs, live, with no preparation. Pay attention to the failure cases as much as the successes. A vendor who refuses this request is a vendor who knows their product won't survive contact with your data.

The question to ask: "Can we run a live test using our actual data right now? I'd like to supply the inputs."

---

Red Flag 2: Vague answers about data privacy, security, and compliance

When you ask an AI vendor about data privacy and security, you should receive specific, documented answers — not reassurances, not deflections, not promises to follow up.

Vendors without comprehensive policies like privacy policies, data processing agreements, and cybersecurity policies signal a lack of preparedness and accountability. Vendors should also be transparent about AI system data flows, subprocessors, model datasets, and training data.

The stakes here are not abstract. A client in the healthcare space once asked a vendor for a Business Associate Agreement before signing. The vendor said they would "have one ready by implementation." Three months later, the client was already sending patient-adjacent data through the platform with no signed BAA. That's a HIPAA violation waiting to be discovered.

In 2026, every credible AI vendor should be able to answer — immediately and specifically — the following questions:

On your data:

• Is our data used to train your AI models?
• Is our data shared with any third parties, including the underlying model providers (OpenAI, Anthropic, Google)?
• Where is our data stored, and in which regions?
• What happens to our data when we cancel the contract?

On security:

• Do you have a SOC 2 Type II report? Can I see it now?
• What is your data breach notification process and timeline?
• Do you conduct regular third-party security audits?

On compliance:

• Are you GDPR compliant? How do you handle data subject rights requests?
• If we are in a regulated industry (healthcare, financial services, legal), what compliance certifications are relevant and how do we access documentation?

If the AI learns from usage — meaning your data improves their model — you need to know that upfront. Is your data used to train their models? Is it shared with third parties? If they're using API-based models from OpenAI, Google, or similar, is your data going to those providers?

Hedged answers, missing documentation, or "we can get that to you later" responses to any of these questions are red flags that carry real regulatory and operational risk. Walk away or put contract language in place that makes the vendor legally accountable before any data flows.

The question to ask: "Can you share your SOC 2 Type II report and your data processing agreement right now, before we continue the evaluation?"

---

Red Flag 3: No clear answer to "what happens when it gets it wrong?"

Every AI system makes mistakes. The outputs are probabilistic, not deterministic. Edge cases will produce wrong answers. Rare inputs will confuse the model. Over time, the distribution of your inputs will shift in ways that reduce accuracy. This is the nature of the technology, not a flaw in any specific product.

The red flag is not that a vendor's AI makes mistakes. The red flag is that the vendor has no clear answer to what happens when it does.

"What happens when the AI gets it wrong?" is one of the most important questions to ask. Everything fails sometimes. How does the system handle mistakes? Can users override it? Is there a human-in-the-loop option? If the vendor claims their AI never makes mistakes, they are lying.

A credible AI vendor will answer this question immediately and specifically because they have designed for it. They will describe:

• Confidence scoring — the system shows users how confident it is in each output, so low-confidence results can be flagged for human review
• Override mechanisms — users can correct wrong outputs, and those corrections improve the system over time
• Audit trails — every AI decision is logged, with the input, output, and confidence level, for review and accountability
• Escalation paths — defined workflows for when the AI should defer to a human rather than produce an output
• Accuracy monitoring — dashboards or alerts that track model performance over time and surface degradation

If the vendor pivots to talking about how accurate their model is, or promises that accuracy will continue to improve, or says their error rate is low enough that it won't matter — none of these answers the question. You're not asking about average accuracy. You're asking about the operational process for handling the cases when it fails.

Plan for AI features to fail sometimes. Have backup processes. Don't create single points of failure where AI breaking means your business stops.

The question to ask: "Walk me through exactly what happens when your AI produces an incorrect output. What does the user see, what can they do, and how does that get corrected?"

---

Red Flag 4: "AI-powered" with no explanation of what that actually means

In 2026, "AI-powered" means nothing on its own. It has become the most overused, least meaningful phrase in business software marketing. Every product from a spreadsheet tool to a legacy CRM has bolted on some form of AI label.

Many products still pitch themselves as "AI-powered" without clearly explaining what problem it solves. Users don't buy AI — they buy outcomes. When AI becomes the headline instead of the tool, products feel vague and interchangeable.

Behind most "AI-powered" claims is one of three realities:

Reality A: The product uses a legitimate AI model (a large language model, a classification model, a computer vision model) that is genuinely integrated into the core workflow and produces the output the marketing claims.

Reality B: The product uses simple rule-based automation or basic pattern matching that has been retroactively labelled as AI because the marketing team noticed that AI sells.

Reality C: The product calls a third-party AI API (usually OpenAI or Anthropic) and passes your inputs to it. The "AI" is entirely someone else's model. The vendor has built a wrapper, not a product. You could replicate the same output by going directly to the API yourself.

Reality C is increasingly common in 2026. It's not necessarily disqualifying — the wrapper might add genuine value through workflow integration, UI, or data handling — but you should understand what you're paying for and whether the pricing reflects the actual work involved.

Questions that reveal which reality you're dealing with:

• "Is the AI model proprietary to your company, or are you using a third-party model?"
• "Which underlying model or models does your product use?"
• "How does your AI component work specifically — is it a language model, a classification model, a recommendation system?"
• "What makes your AI implementation differentiated from a direct integration with the underlying model?"

If the sales team cannot explain what their AI does in concrete terms — not "it learns from your data" or "it uses advanced machine learning algorithms" but the actual mechanism, the actual input, the actual output — you are likely dealing with Reality B or a particularly evasive version of Reality C.

The best AI tools are often the ones you barely notice. They quietly make something easier or faster without requiring you to become an AI expert. But that ease should come from design, not from the vendor not understanding their own product well enough to explain it.

The question to ask: "Explain exactly what happens technically when I input data into your AI feature. What model is running, what does it produce, and what makes it specific to your product rather than a generic model call?"

---

Red Flag 5: Pricing and contract terms designed to lock you in before you prove value

The final red flag is in the contract, not the product — and it is where the most expensive mistakes happen.

AI software vendors who are confident in their product's value will structure contracts that let you prove that value before committing fully. Vendors who are not confident will structure contracts that lock you in before you have evidence either way.

Many rigid off-the-shelf AI systems trap businesses with inflexible pricing models and no escape route when the vendor falls behind AI's rapid evolution.

The specific contract patterns to watch for:

Long annual commitments with no exit clause. Requiring a 12-month contract for a new AI tool before you've run a proper pilot is a significant risk. What if the tool doesn't perform as promised on your real data? What if the vendor's underlying model changes and performance degrades? What if a better tool emerges in six months — which in the AI market of 2026 is not a hypothetical? Negotiate for a 90-day pilot with an explicit exit option if defined performance benchmarks aren't met.

Usage-based pricing with no caps. AI APIs charge per token, per query, or per document. Usage-based pricing is reasonable in principle but dangerous without caps. A single runaway process, a loop in an agentic workflow, or a sudden traffic spike can produce a bill orders of magnitude higher than projected. Require hard monthly spending caps and alerts before you approach them.

Vague "enhancement" clauses. Many AI vendor contracts include language allowing them to change the underlying model, retrain on your data, or modify the system's behaviour in ways described as "improvements." Your signed contract might not bind them to the performance you evaluated. Require explicit contract language specifying that material changes to the AI system require your consent and a re-evaluation period.

Data ownership ambiguity. Some vendor contracts include clauses granting them broad rights to use your data for model training, product improvement, or benchmarking. Read the data ownership section of every AI software contract carefully, with legal counsel if the data is sensitive. Ensure the contract explicitly states that you own your data and have the right to export or delete it at any time.

Hidden implementation costs. The licence fee is only part of the cost. Provide feedback to improve the system. If the AI learns from usage, make sure you're correcting its mistakes and reinforcing good outputs. Passive use won't optimise the system. Before you sign, get a written breakdown of implementation costs, training costs, ongoing support costs, and the cost of any customisation your use case requires. Compare the total first-year cost — not just the licence — to your expected ROI.

The question to ask: "Can we start with a 90-day pilot at reduced commitment, with defined performance metrics that trigger the full contract if met? What is the total cost including implementation, training, and support for the first 12 months?"

---

How to run a proper AI software evaluation in 2026

The five red flags above tell you what to watch for. Here's the evaluation process that surfaces them:

Week 1 — Define success before you engage any vendor

Before a single demo, write down: what specific problem are you solving, what does success look like in measurable terms (time saved, error rate reduced, tickets resolved), and what data will you use to test the product. This definition anchors every subsequent conversation.

Week 2 — Run vendor demos with your own data

Contact your shortlist of vendors and request live demonstrations using your data. Provide your test inputs in advance (so they can't claim technical setup is needed) but require the demo to run on your actual data. Evaluate outputs on your edge cases, not their showcase examples.

Week 3 — Conduct security and compliance due diligence

Request SOC 2 reports, data processing agreements, and privacy policies. If you're in a regulated industry, involve your legal and compliance team before the evaluation progresses. Any vendor who can't provide documentation within a week should be removed from consideration.

Week 4 — Negotiate the pilot terms

For any vendor that passes the technical and compliance evaluation, negotiate a paid pilot (90 days or less) with defined success metrics. Pay a proportionally reduced fee for the pilot period. Include an explicit exit clause if metrics aren't met by the end of the pilot.

Post-pilot — Evaluate against your baseline

Compare the AI tool's performance against the baseline you measured in Week 1. Did it actually reduce the time, error rate, or cost you projected? Only commit to the full contract if the data supports it — not if the vendor's customer success team is enthusiastic.

---

The questions to ask every AI vendor before you sign

Print this list and bring it to every vendor meeting:

Question	What it reveals
Can we run a live test with our data right now?	Whether the product actually works on real data
Do you have a SOC 2 Type II report available today?	Security and compliance readiness
What happens when your AI produces a wrong output?	Error handling design and operational maturity
Which underlying model does your product use?	Whether it's proprietary or a third-party wrapper
Is our data used to train your models?	Data privacy and usage practices
Can we start with a 90-day pilot with an exit clause?	Vendor confidence in their product
What is the total first-year cost including implementation?	True cost of ownership
How do you notify customers of material changes to the AI system?	Contract stability and change management

---

Frequently asked questions

How do I know if an AI tool is genuinely using AI or just basic automation? Ask the vendor to explain what happens technically when you submit an input. A genuine AI implementation will involve a model — they should be able to name it, describe what it produces, and explain what makes it specific to their product. If the explanation sounds like rule-based logic ("if the input contains X, the system returns Y"), you're likely dealing with basic automation marketed as AI. If they can't explain the mechanism at all, that's a more serious concern.

What is the biggest mistake businesses make when buying AI software? Nearly nine in ten buyers who regret a software purchase first encountered an unexpected disruption during implementation. The root cause in almost every case is skipping a proper pilot with real data and real workflows before signing a full contract. The second biggest mistake is not defining success metrics upfront — without a clear baseline and a clear target, you can't evaluate whether the tool is actually working.

Should I avoid AI software with usage-based pricing? Not necessarily — usage-based pricing aligns the vendor's incentives with your usage and can be more cost-effective than flat fees at lower volumes. The danger is uncapped usage-based pricing with no monitoring. Negotiate hard spending caps, require email or SMS alerts when you approach those caps, and monitor costs weekly in the early months of any AI tool deployment.

What should I look for in an AI vendor's data privacy policy? The key questions: Does the vendor use your data to train their models? Is your data shared with subprocessors or underlying model providers? Where is data stored geographically? What is the data retention policy and how do you request deletion? What is the breach notification process? Any policy that is vague on these points, or that grants the vendor broad rights to use your data for their own improvement, should be reviewed carefully by legal counsel before signing.

Is it reasonable to ask for a pilot before committing to an annual contract? Completely reasonable — and increasingly standard for credible AI vendors in 2026. Any vendor who refuses a pilot or makes it structurally difficult (requiring full-price payment, no exit clause, or excessive minimum commitment for the "pilot" period) is telling you something important about their confidence in the product's performance on real customer data.

How quickly is the AI software market changing in 2026? Fast enough that long-term vendor lock-in is a genuine strategic risk. A tool that is best-in-class today may be significantly behind the market in 18 months as the underlying models improve rapidly. Negotiate contract terms that allow you to exit or renegotiate if the vendor's AI performance falls materially behind market alternatives. Build your procurement strategy around flexibility, not commitment length.