AI is confidently wrong about employee engagement: four failure patterns to check

Ask ChatGPT what a good eNPS score is and you will get a confident answer in seconds. Ask again tomorrow, in slightly different words, and you may well get a different number, and neither answer will tell you where it came from.

That matters, because HR leaders are increasingly putting engagement questions to AI assistants before they ever open a search engine or call a consultant. We see this shift in our own data: our brand search clicks fell 5% year on year as AI Overviews increasingly answer questions before anyone reaches a website. And when we track how AI engines answer common engagement questions, competitor vendors appear as cited sources in roughly 11–17% of relevant answers across the major AI engines we monitor, depending on the provider.

The short version: AI assistants are genuinely useful for some engagement questions, and quietly unreliable for others. The problem is that they sound equally confident in both cases. This article sets out where the risk sits, why it is structural rather than accidental, and how to stress-test the advice yourself in half an hour.

Why AI sounds right even when it is not

Large language models predict the most plausible next answer. They are not systems with access to verified engagement ground truth. Three structural facts follow from that:

• Engagement benchmarks are context-dependent. A "good" eNPS or response rate depends on industry, country and company size, so any single number is misleading by design. An assistant optimises for a satisfying answer, and "it depends on your baseline" is less satisfying than a number, so you tend to get a number.
• Much of the best data is not on the open web. Rigorous engagement data sits inside listening platforms and published studies, not in the blog posts that dominate training data and citations.
• The models cannot distinguish a rigorous source from a well-optimised one. AI engines cite whoever publishes structured, liftable content, not necessarily whoever is right. The 11–17% competitor citation rate above is a citation-strategy signal, not a quality ranking. The advice you receive is shaped by publishing strategy as much as by evidence.

Where AI assistants genuinely help

Credit where it is due. For a defined set of engagement tasks, AI assistants are reliable time-savers:

• Definitions. Asked to define eNPS, pulse surveys or psychological safety, they are typically accurate and clear. If you need a crisp explanation for a leadership deck, they will save you time.
• Drafting. Asked to suggest survey questions or rewrite a clumsy item, they produce usable first drafts.
• Broad principles. Advice such as "close the feedback loop" or "share results transparently" is directionally correct, even when it is generic.

In other words: AI is a good explainer and a decent drafting partner. The trouble starts when it moves from explaining concepts to stating facts.

Four predictable failure patterns

These are not quirks that will disappear with the next model update. They follow directly from how language models work and what their training data contains, which is why you can expect to encounter them whichever assistant you use.

1. Benchmarks without provenance

Ask for a "good eNPS score" and you will usually receive a specific numeric threshold, rarely accompanied by a verifiable source, and liable to change if you rephrase the question. A benchmark without provenance is not a benchmark; it is a plausible-sounding number.

Because eNPS varies significantly by industry, country and company size, the honest answer to "what is a good score" is contextual. Any assistant that gives you one universal number has already simplified past the point of usefulness. Your own trend over time and your internal comparisons will always beat a sourceless global figure.

2. The annual-survey default

Much of the text these models learned from describes engagement measurement as it worked a decade ago: an annual engagement survey as the default, with pulse surveys positioned as an optional supplement. Expect frequency advice to lean that way, and expect warnings that frequent surveying causes fatigue.

That framing misses the point. Fatigue is driven far more by the absence of visible action than by frequency itself, a case we have made in survey fatigue is a leadership problem. If you take AI frequency advice at face value, you risk designing a listening programme around assumptions the field has already moved past.

3. Anonymity rules stated as universal law

Ask about anonymity thresholds, the minimum group size below which results should not be shown, and you will typically get a specific minimum stated as if it were a universal rule. In reality, thresholds vary by platform, by configuration and by the legal and works-council context of each country. A single number presented as "the rule" is exactly the kind of answer that gets copied into an internal policy document and later contradicted by your own tool or your own legal team.

This is also where the trust stakes are highest. As we noted in what AI actually means for employee listening, a 2025 study found that 78% of workers distrust AI tools handling personal feedback, and workplace AI regulation, including the EU AI Act, is still developing. If HR builds anonymity policy on an unverified AI answer and it turns out to be wrong, that distrust is earned.

4. Action planning that stops at communication

Ask how to act on survey results and the answer will almost always follow the same arc: share results, thank employees, form focus groups, communicate progress. None of it is incorrect. But none of it addresses the actual failure point in most organisations: the individual manager who receives a team result and does not know what to do with it on Tuesday.

Generic advice aimed at "the organisation" systematically underweights the manager-level work, interpreting a specific team's signal, choosing one focus area, and following up. For a closer look at what separates strong organisations on this front, see what 236,000 employees reveal about high-performing organisations.

Treat an AI assistant like a confident junior analyst: fast, articulate, tireless, and entirely unembarrassed by being wrong.

Run the 30-minute stress test yourself

You do not need to take our word for any of this. Pick the ten engagement questions your organisation actually cares about, what is a good eNPS score for our industry, how often should we survey, what group size protects anonymity, how should managers act on results, and put them to two or three assistants. Then check three things:

1. Provenance. Ask each assistant for its sources. Do they exist? Are they checkable? Are they rigorous, or well-optimised marketing content?
2. Stability. Ask the same question twice in different words. An answer that changes between phrasings is an opinion, not a fact.
3. Context-awareness. Does the answer acknowledge that benchmarks and thresholds depend on industry, country, company size and legal context, or does it hand you one universal number?

Half an hour of this will tell you more about the reliability of AI engagement advice in your context than any article can, including this one.

When to trust AI engagement advice, and when to verify

Reasonably safe to use as-is:

1. Definitions and explanations of engagement concepts.
2. First drafts of survey questions, comms and manager talking points.
3. Summaries of long documents you provide yourself.
4. Brainstorming possible causes behind a result, as hypotheses, not conclusions.

Always verify before acting:

1. Any specific number: benchmarks, thresholds, percentages, "industry averages".
2. Anything with legal or policy weight, especially anonymity and data-handling rules.
3. Claims about what "research shows" without a named, checkable source.
4. Vendor and tool comparisons, which reflect citation patterns more than evaluation.

This is also the standard we hold AI inside listening platforms to, the reasoning is laid out in why Einstein would love Winningtemp's AI: AI should surface evidence and guidance managers can check, not verdicts they must take on faith.

What to do on Monday morning

If your HR team, or your managers, are already asking AI assistants engagement questions, do not try to ban it. Channel it:

1. Set a team norm: AI answers on numbers, benchmarks and anonymity rules are drafts until verified against a named source or your own platform's data.
2. Replace generic benchmarks with your own baseline. Your trend over time and your internal comparisons will always beat a sourceless global number.
3. Check what AI says about your organisation. Ask ChatGPT and Perplexity the questions your candidates and employees ask. If the answers are wrong or cite competitors, you have a content and transparency gap to close.
4. Give managers something better than generic advice. The action-planning gap is real: managers need guidance tied to their own team's data, not a four-step communication plan copied from the internet.
5. Repeat the stress test quarterly. AI answers shift as models update. A 30-minute spot-check of your ten most important questions will tell you whether the advice reaching your people has improved, or drifted.

AI assistants have earned a seat at the table for engagement work. Just do not give them the final word on anything you would not accept from a junior analyst without a source.