Myth vs. Evidence: What Vendors Say About AI — and What the Data Actually Shows

How to Use This Card

Executives hear AI claims daily — from vendors, consultants, board members, and employees. Most are directionally true but materially misleading. This card pairs ten common claims with the independent evidence, so the next pitch deck lands on an informed audience.

The Ten Claims

1. “AI makes developers 55% more productive”

The claim: GitHub’s original Copilot study (Peng et al., 2023) found developers completed a task 55.8% faster with AI assistance. This number appears in nearly every vendor deck.

What the evidence shows: That study used 95 freelancers on a single greenfield JavaScript task with no code review, no integration, and a 63% dropout rate. Six subsequent studies paint a different picture. Google’s own RCT (n=96 engineers, October 2024) found 21% — with a confidence interval so wide it includes near-zero. Stanford’s analysis of 100,000 developers across 600+ companies finds the real average is 15-20%, collapsing to 0-10% on complex work in existing codebases. The largest peer-reviewed study (Science, 2025, 170,000 developers) found 3.6% quarterly output growth. METR’s independent RCT (n=16, 246 tasks, July 2025) found experienced developers were 19% slower when using AI indiscriminately.

The takeaway: Budget for 10-20% gains on routine tasks. If your business case requires 40%+ to justify the investment, it will not pencil out.

2. “AI closes the skill gap between junior and senior developers”

The claim: Junior developers benefit most from AI, leveling the playing field with experienced engineers.

What the evidence shows: Two credible studies reach opposite conclusions. Cui et al. (n=4,867 developers, three RCTs, Management Science, 2026) found juniors gained 21-40% in task completion while seniors gained 7-16%. But the largest observational study — Daniotti et al. (170,000 developers, 30 million commits, Science, 2025) — found senior developers capture nearly all productivity and exploration gains while juniors show no statistically significant benefit.

The takeaway: The answer depends on what you measure. Juniors complete more tasks; seniors produce code that survives in production. Both findings are credible. Do not restructure your workforce based on either one alone.

3. “AI adoption pays for itself with the license fee”

The claim: At $19-39/seat/month, AI coding tools are the cheapest productivity investment available.

What the evidence shows: The license is 4.4% of the true cost. A 10-person team pays $8,400/year in subscriptions — and $192,000/year in debugging AI-generated code, additional code review time, governance overhead, and training. Faros AI tracked 10,000+ developers across 1,255 teams and found AI produces 98% more pull requests but 91% more review time. The speed gain evaporates unless you redesign the review process. CodeRabbit’s independent analysis found AI-generated code creates 1.7x more downstream issues.

The takeaway: License fees represent 10-20% of the total investment. Organizations that budget for the full cost upfront — 2.5x the license in Year 1 (DX Research/Atlan, 2025) — survive the CFO’s first ROI review. Those that don’t see the program defunded at month six.

4. “95% of companies are adopting AI — you’ll fall behind”

The claim: Adoption is universal. Not having an AI strategy is a competitive risk.

What the evidence shows: Adoption is high — McKinsey (n=1,993, July 2025) finds 88% use AI in at least one function. But adoption and value are different things. Only 6% of organizations show measurable EBIT impact above 5% (McKinsey). 60% generate no material value (BCG, n=1,250, September 2025). 42% of companies abandoned most of their AI initiatives in 2025, up from 17% in 2024 (S&P Global, n=1,006). MIT’s research puts the generative AI pilot failure rate at 95%.

The takeaway: The risk is not falling behind on adoption. The risk is adopting without a plan and joining the 42% that spent the money and abandoned the work. The 5% that capture value do three things differently: deploy on the right tasks, redesign workflows, and define success metrics before spending.

5. “AI will eliminate 30-40% of jobs”

The claim: Mass displacement is coming. Companies need to prepare for dramatic headcount reduction.

What the evidence shows: Among companies experiencing AI productivity gains, only 17% reduced headcount (EY, n=500 SVP+ leaders, October 2025). 83% reinvested gains into growth, R&D, and upskilling. The companies that cut aggressively became cautionary tales: Klarna eliminated 40% of its workforce, then publicly reversed course — CEO Sebastian Siemiatkowski stated “we went too far.” IKEA retrained 8,500 call center workers as design advisors, generating $1.4 billion in new revenue with zero layoffs. Challenger, Gray & Christmas tracked 54,836 AI-cited job cuts in 2025 — roughly 4.5% of all announced layoffs that year.

The takeaway: AI reshapes roles, it does not eliminate them at the scale headlines suggest. The companies producing measurable value augment and redeploy. The companies that cut headcount as a primary strategy either reversed course or accelerated their decline.

6. “Just run a few pilots and scale what works”

The claim: Start small, experiment broadly, and the best use cases will reveal themselves.

What the evidence shows: The average organization scraps 46% of proofs-of-concept before production (S&P Global, n=1,006, 2025). Fortune documented companies running 30-50+ simultaneous pilots — one healthcare company had over 900. The problem: pilots designed to answer “does AI work?” rarely include security review, integration architecture, cost modeling, or kill criteria. Production deployment costs 3-5x the pilot budget. MIT found 95% of generative AI pilots fail to deliver measurable P&L impact. Pertama Partners (n=2,400+ initiatives) found projects with pre-defined success metrics achieve 54% success versus 12% without.

The takeaway: Fewer pilots with production paths beat more pilots without them. The discipline is not launching experiments. It is launching experiments designed to become operations.

7. “Our employees love AI — adoption is at 80%”

The claim: High usage metrics prove the AI investment is working.

What the evidence shows: BCG (n=10,635 employees, 2025) found 76% of executives believe employees are enthusiastic about AI — but only 31% of individual contributors actually are. Microsoft’s Dear Diary RCT (n=228 engineers) found developers reported feeling more productive with AI, but telemetry showed no measurable difference in output. METR documented a 39-percentage-point perception gap: developers predicted a 24% speedup and estimated a 20% gain afterward — while actually working 19% slower. Writer/Workplace Intelligence (n=1,600, March 2025) found 31% of employees actively sabotage their company’s AI strategy.

The takeaway: Usage is not value. Satisfaction surveys are not ROI. Measure pull request cycle time, defect rates, time-to-production, and rework rates. If the tool is working, those numbers move. If only the survey numbers move, the tool is not working.

8. “AI is plug-and-play — just turn it on”

The claim: Modern AI tools require minimal configuration. Activate the license and productivity flows.

What the evidence shows: Deloitte (n=3,235 senior leaders, 2025) finds 37% of organizations use AI at a surface level with no process changes. Only 30% redesigned workflows. ActivTrak’s behavioral study (n=163,638 workers, 443 million hours) found that after AI deployment, no work category decreased — email volume rose 104%, chat messages rose 145%, while deep-focus sessions dropped 9%. AI added speed to individual steps without subtracting work from the system. McKinsey finds organizations that capture value allocate 10% to algorithms, 20% to technology and data, and 70% to people and process change. BCG confirms: companies with this ratio achieve 1.7x revenue growth and 3.6x total shareholder return.

The takeaway: The tool is the easy part. Workflow redesign is 70% of the value. If you activate the license without changing how work flows, you will measure individual speed gains and organizational stagnation.

9. “Your data is safe with our AI platform”

The claim: Enterprise AI tools protect your data through SOC 2, encryption, and tenant isolation.

What the evidence shows: 77% of employees paste company data into AI tools through personal accounts the organization cannot see (LayerX, October 2025). 82% use personal accounts. Shadow AI breaches cost $670,000 more per incident than standard breaches — $4.63M versus $3.96M (IBM Cost of a Data Breach, 2025, n=604 organizations). Only 37% of organizations have any AI governance policies. 98% report unsanctioned AI use (Varonis, 2025). In 2025, security researchers discovered 225,000+ OpenAI credentials for sale on dark web markets with full chat histories — every prompt, every pasted document, every client name.

The takeaway: The enterprise platform may be secure. The problem is the 200+ employees using personal ChatGPT accounts to process your client data outside it. A shadow AI audit takes 30 days and typically reveals 3-5x the expected tool footprint.

10. “AI delivers ROI in the first quarter”

The claim: Fast time-to-value — deploy AI and see returns within 90 days.

What the evidence shows: 89% of managers report zero AI productivity impact over the past three years (NBER, n=5,956 executives, February 2026). Only 14% of CFOs see measurable AI ROI (RGP, n=200 CFOs, October-November 2025). The median abandoned AI project consumed 11 months and $4.2 million before termination (Pertama Partners, n=2,400+ initiatives). Targeted automation tells a different story: AP invoice processing delivers 80% cost reduction with 3-6 month payback (APQC benchmarks). Customer service routing shows 14% productivity gains in a peer-reviewed RCT (Stanford/MIT, n=5,179). The difference is specificity — boring automation against a named bottleneck versus open-ended “AI strategy.”

The takeaway: Generative AI deployed broadly produces no measurable ROI in the timeline vendors promise. Targeted automation against a specific, measurable process delivers returns in months. The path to ROI is not an AI strategy — it is an operations audit that identifies your most expensive manual process and automates it.

Key Data Points

What This Means for Your Organization

This card is a filter. The next time a vendor, consultant, or board member presents an AI claim, the question is not whether the claim is true — most are, directionally. The question is whether the evidence supports the magnitude and the timeline being proposed, and whether your organization has the workflow redesign capacity to capture the gains.

The 5% of organizations producing measurable AI returns are not more technically sophisticated. They are more honest about what the data shows and more disciplined about acting on it — deploying AI against specific bottlenecks, measuring outcomes that connect to the P&L, and redesigning workflows rather than layering tools onto broken processes.

If this card raised questions about how the evidence applies to your specific situation — or if you want help running the shadow AI audit or operations assessment that precedes any credible AI investment — I would welcome the conversation at brandon@brandonsneider.com.

Sources

1. Peng et al. — “The Impact of AI on Developer Productivity.” arXiv:2302.06590, February 2023. n=95. GitHub-commissioned. https://arxiv.org/abs/2302.06590
2. Paradis et al. — Google internal RCT. arXiv:2410.12944, October 2024. n=96. https://arxiv.org/abs/2410.12944
3. Denisov-Blanch — Stanford, ~100K developers, 600+ companies. arXiv:2409.15152, 2025. https://softwareengineeringproductivity.stanford.edu/ai-impact
4. Daniotti et al. — Science, 2025. 170K developers, 30M commits. DOI: 10.1126/science.adz9311
5. Becker et al. / METR — arXiv:2507.09089, July 2025. n=16, 246 tasks. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
6. METR update — February 2026. n=57, 800+ tasks. https://metr.org/blog/2026-02-24-uplift-update/
7. Cui et al. — Management Science, 2026. n=4,867. https://pubsonline.informs.org/doi/abs/10.1287/mnsc.2025.00535
8. Butler et al. — Microsoft Dear Diary. arXiv:2410.18334, October 2024. n=228. https://arxiv.org/abs/2410.18334
9. Faros AI / AlterSquare — 10,000+ developers, 1,255 teams. 2025. https://www.faros.ai
10. McKinsey — State of AI 2025. n=1,993. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
11. BCG — Build for the Future. n=1,250, September 2025. https://www.bcg.com/publications/2025/closing-the-ai-impact-gap
12. BCG — AI at Work. n=10,635, June 2025. https://www.bcg.com/publications/2025/ai-at-work-momentum-builds-but-gaps-remain
13. S&P Global 451 Research — n=1,006, March 2025. https://www.spglobal.com/market-intelligence/en/news-insights/research/ai-experiences-rapid-adoption-but-with-mixed-outcomes-highlights-from-vote-ai-machine-learning
14. DX Research/Atlan — Year 1 TCO analysis. License fees = 10-20% of total AI deployment cost. 2.5x Year 1 multiplier. 2025.
15. Pertama Partners — n=2,400+ initiatives, 2025. https://www.pertamapartners.com/insights/ai-project-failure-statistics-2026
16. Deloitte — State of AI 2026. n=3,235. https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html
17. ActivTrak — n=163,638 workers, 443M hours, 2025. https://www.activtrak.com/resources/research/
18. EY — AI Pulse Survey. n=500 SVP+ leaders, October 2025.
19. Challenger, Gray & Christmas — AI-cited job cuts tracking, 2023-2026.
20. NBER Working Paper 34836 — n=5,956 executives, February 2026. https://www.nber.org/papers/w34836
21. RGP — n=200 CFOs, October-November 2025.
22. LayerX — Enterprise AI & SaaS Data Security Report, October 2025.
23. IBM — Cost of a Data Breach 2025. n=604 organizations. https://www.ibm.com/reports/data-breach
24. Writer/Workplace Intelligence — n=1,600, March 2025. https://writer.com/resources/ai-at-work-research/
25. APQC — AP Benchmarks 2025. https://www.apqc.org/resources/benchmarking/open-standards-benchmarking/measures/total-cost-perform-process-process-19
26. Brynjolfsson, Li & Raymond — QJE, 2025. n=5,179. https://academic.oup.com/qje/article/140/2/889/7990658

Brandon Sneider | brandon@brandonsneider.com March 2026