AI-powered CX insights promise speed, scale and innovation. Yet when we tested these claims in practice, the reality contradicted the hype: fast outputs, yes — but with major risks to quality and rigour.
Our experiment put AI synthesis to the test using qualitative research transcripts. The AI-generated insights were fast but often shallow, missing the contextual nuance, innovation signals, and strategic framing leaders need for robust decisions.
Over two years, I’ve been experimenting with both specialist CX platforms and general-purpose large language models (LLMs). The aim: understand where AI augments qualitative research — and where its limitations erode decision-making confidence.
At the AI for Researchers workshop, led by Dr Llewyn Paine, 30 researchers ran the same AI-assisted synthesis experiment. We tested transcripts simultaneously, controlling parameters and prompts, to compare outputs for accuracy, coverage and originality.
The findings were clear: AI-generated synthesis suffers from inconsistency (different outputs from identical inputs), weak rigour (thin evidence chains), inaccuracies (unsupported themes, misattributed quotes), and position-bias (overweighting content from the start of transcripts).
The biggest issue? Rigour. AI often presented a single participant’s comment as “evidence” for a theme. This overstates prevalence and hides nuance — a fundamental risk when leaders use insights to guide strategy.
For executives, the message is caution and care. AI can accelerate parts of the insight workflow, but must be treated as a junior analyst: clear instructions, chunked workflows and close human supervision are non-negotiable.
The value lies in combining speed with governance. Leaders can capture efficiency gains only when AI is paired with human validation, transparent prompt design and evidence-backed synthesis that withstands strategic scrutiny.
The opportunity is real — but so are the risks. Used responsibly, AI can unlock innovation and improve customer experiences. Without oversight, it risks eroding the very insights leaders rely on for growth.
My takeaway: AI-powered customer journey intelligence is coming — but for now, leaders must balance speed with strategic value. Proceed with care, invest in oversight, and keep insight quality non-negotiable.
BIG Thanks to Dr Dr Llewyn Paine.
See Rosenfeld Media for Dr Llewyn Paine’s ‘AI for UX Researchers’: 2-day virtual workshop.
Read about the AI-generated insights experiment findings here: