The 5% Problem: What 1.4 Million Real AI Interactions Reveal About Sophisticated Use at Work

See also (wiki): productivity-rcts, training-architecture, ai-change-management, functional-manager-ai-adoption

Executive Summary

• A joint University of Texas at Austin / KPMG study (n=2,597 users, 1.4 million real workplace AI interactions, 8 months of behavioral data, published HBR March 2026) finds that only 5% of employees use AI with genuine sophistication — despite approximately 90% using AI regularly.
• The 5% are not distinguished by technical skill or prompt-engineering expertise. They are distinguished by four observable behavioral patterns: they return to AI repeatedly, they iterate rather than accept first outputs, they make ambitious initial requests, and they deliberately choose which tool to use for which task.
• The core differentiator is treating AI as a reasoning partner — asking it to assume roles, verify its own reasoning, and refine across multiple exchanges — versus treating it as an answer machine that responds to single queries.
• Most organizations measure AI adoption by login frequency and seat utilization. Neither metric captures behavioral sophistication. This is why adoption dashboards look healthy while ROI remains elusive.
• The research runs counter to the vendor interest of KPMG, which stands to sell more AI training if adoption looks successful. The “5% sophisticated” finding, sourced from KPMG’s own employee data, is more credible because of that tension.

The Study

Researchers from the McCombs School of Business at the University of Texas at Austin, partnering with KPMG LLP, analyzed 1.4 million real workplace AI interactions from 2,597 unique users over eight months of KPMG’s back-office operations. They evaluated more than 30 behavioral characteristics embedded in actual AI conversations — task complexity, prompting technique, iteration patterns, tool selection — without relying on surveys or self-report.

The result was a behavioral fingerprint of sophisticated AI use, published in Harvard Business Review on March 19, 2026.

The dataset spans an unusually wide range of behavior: first prompts ranged from 26 to 48,670 characters; iteration ranged from 1 to 45 exchanges per conversation; users engaged anywhere from 1 to 3 different AI models. The variation within a single organization, across people with the same tools and the same formal AI access, was dramatic.

Credibility note: This is behavioral data from real interactions, not a survey measuring self-reported use. Academic authorship (UT Austin McCombs) introduces peer-review discipline. KPMG co-authorship introduces consulting-firm caveat — KPMG sells AI training services. However, the central finding (only 5% of their own employees use AI well) runs counter to that commercial interest, which strengthens the result’s credibility.

What the 5% Do Differently

The researchers identified four measurable behavioral signals that consistently distinguish high-impact AI users from routine users. These are not about prompts — they are about engagement patterns across time.

1. Return frequency. Sophisticated users come back to AI tools repeatedly for the same problem, treating a session as one exchange in an ongoing dialogue rather than a one-shot query.

2. Persistence in refinement. Rather than accepting an initial output, high-impact users iterate — requesting alternatives, asking for verification, pushing back on reasoning, demanding different structure. The first response is a starting point, not a deliverable.

3. Ambition level. Sophisticated users make demanding initial requests: multi-step tasks with specified constraints, complex analytical problems, synthesis across multiple inputs. They do not ask the AI to summarize a document when they could ask it to identify the three decision implications of that document and stress-test each.

4. Intentionality in tool selection. The top 5% select which AI model or tool to use based on the task. They do not default to one tool for everything.

The underlying behavior these four signals measure is the same: treating AI as a general cognitive tool rather than a narrow productivity aid. The researchers’ summary: “The most sophisticated users don’t ‘prompt better’ — they work better with AI.”

One counterintuitive finding: sophisticated users often use informal, conversational language in their prompts. The sophistication is not in formal phrasing — it is in the structure of the problem they hand to the AI, the constraints they specify, and the verification they demand.

Why This Matters to Organizations

The METR RCT (July 2025, n=16 experienced developers, 246 tasks) found that average developers using AI on open-ended tasks were 19% slower than without AI, while perceiving themselves as 20% faster. The gap between perception and reality reached 39 percentage points. The standard explanation is that AI assistance degrades expert judgment or creates overconfidence.

The UT Austin / KPMG data offers a complementary explanation: most people using AI are not using it the way the 5% do. The average user treats AI as an answer machine. The expert user treats it as a thinking partner. The behavioral gap between those two modes of use is large enough to produce meaningfully different outcomes — and neither the METR population nor the broader KPMG population appears to have trained into sophisticated use naturally.

This matters for several related findings in the 2026 research corpus:

• BCG’s “AI at Work 2025” (n=10,635) finds only 5% of organizations achieve substantial financial gains from AI — the same 5% figure, now appearing at the organizational level.
• The DataCamp/YouGov study (n=517, Feb 2026) finds organizations with mature training programs double their significant-ROI rate (21% → 42%). But “mature” means role-tailored content, hands-on labs, and measured outcomes — not seat licenses and video courses.
• The Gallup Q1 2026 tracking study (n=23,717) finds that employees whose managers actively champion AI use are 79% vs. 46% frequent AI users. Behavioral modeling from leadership produces measurable behavioral change downstream.

The pattern across these studies is consistent: access to AI tools is not the constraint. Behavioral sophistication — how employees actually engage with the tools they already have — is the constraint.

Key Data Points

What This Means for Your Organization

The immediate implication is diagnostic: if your AI adoption dashboard shows high login rates and seat utilization but modest financial returns, the most likely explanation is not tool quality or access. It is behavioral sophistication at scale.

The four behavioral signals the researchers identified are measurable in your own environment today. Prompt length distribution, iteration rates per conversation, tool-switching patterns, and task complexity indicators are all observable in enterprise AI platform logs. Before assuming a training gap, check whether your current platform exposes these signals. If it does not, the vendor conversation becomes about data access, not just feature roadmap.

The training implication follows: role-specific behavioral modeling produces measurably different outcomes than vendor-provided prompt-engineering courses. KPMG’s response was to build role-aligned learning that translated the four behavioral patterns into daily practice — with peer-led champions, not just video libraries. The DataCamp/YouGov data provides the financial case: organizations with mature programs achieve double the significant-ROI rate, and the defining characteristic of “mature” is hands-on, role-tailored, measured learning — not seat volume.

If these findings raise questions specific to your organization’s AI training program design or behavioral measurement approach, a direct conversation is often the fastest path to clarity — brandon@brandonsneider.com.

Sources

1. UT Austin McCombs School of Business / KPMG LLP — “Behaviors Behind High-Impact AI Use” (Hallman, Kowaleski, Schmidt, Puvvada). Published: Harvard Business Review, March 19, 2026. URL: https://hbr.org/2026/03/what-the-best-ai-users-do-differently-and-how-to-level-up-all-of-your-employees. KPMG press release: https://kpmg.com/us/en/media/news/utaustin-kpmg-study.html. Credibility: HIGH for behavioral data methodology (1.4M real interactions, not survey); MEDIUM-HIGH for overall study (consulting-firm co-authorship; “5% sophisticated” finding runs counter to KPMG’s commercial interest which increases credibility; academic peer-review track through HBR editorial process). Note: KPMG sells AI training services. Apply consulting-firm caveat. However, the primary dataset is behavioral, not attitudinal, and the headline finding is unflattering to KPMG’s own adoption program.

2. BCG / MIT Sloan “AI at Work 2025” (n=10,635, 11 countries). URL: https://www.bcg.com/publications/2025/ai-at-work-momentum-builds-but-gaps-remain. Credibility: MEDIUM-HIGH (large sample, independent survey firm; BCG vendor caveat for consulting).

3. DataCamp / YouGov “2026 State of Data & AI Literacy Report” (n=517 US and UK enterprise leaders, YouGov independent fieldwork, published February 26, 2026). URL: https://www.datacamp.com/blog/ai-roi-in-2026-why-workforce-capability-determines-the-return-on-ai. Credibility: MEDIUM-HIGH (YouGov independent fieldwork; DataCamp vendor caveat).

4. Gallup Q1 2026 AI Workplace Tracking (n=23,717, Feb 2026, ±0.9pp). Credibility: HIGH (Gallup Panel, probability-based, non-commercial).

Brandon Sneider | brandon@brandonsneider.com April 2026