← Adoption Challenges 10 min read

How Fortune 500 Companies Structure AI Centers of Excellence

Executive Summary

  • Hub-and-spoke is the winning model. IBM’s 2025 survey of 600+ CAIOs finds centralized and hub-and-spoke AI organizations deliver 36% higher ROI than decentralized structures — yet most enterprises still lack one. Only 14% of Fortune 500 executives say they are fully ready for AI deployment (Sedgwick 2026, n=300).
  • The Chief AI Officer is going mainstream, fast. CAIO adoption grew from 11% of organizations in 2023 to 26% in 2025, with 40% of Fortune 500 companies projected to have one by 2026. Organizations with CAIOs see 10% higher ROI on AI spend (IBM IBV 2025, n=600+).
  • Governance structures exist on paper but not in practice. 70% of Fortune 500 executives report having AI risk committees, but only 28% of S&P 100 companies disclose both board-level oversight and a formal AI policy (Harvard Law Forum analysis, 2025). The gap between governance theater and operational reality is the central challenge.
  • Most AI projects still fail. Only 25% of AI initiatives have delivered expected ROI, and only 16% have scaled enterprise-wide (IBM 2025). MIT’s State of AI in Business 2025 report puts the production success rate at 5%. AI Centers of Excellence are the primary organizational response, but structure alone does not fix execution.
  • The biggest emerging demand is for AI governance skills, not technical AI skills. Fortune 500 job postings for AI governance and model risk grew 81% year-over-year, versus 30% growth for individual contributor AI roles (Draup 2026).

Three Operating Models — and Why One Wins

Enterprises structure their AI operations around three models. The choice determines whether AI remains a science project or becomes an operating capability.

Centralized CoE

A single team owns all AI strategy, governance, talent, and execution. This works well for organizations in early AI maturity — it prevents fragmentation and builds institutional knowledge. McKinsey’s 2025 State of AI survey (n=1,993, 105 nations) finds that governance-heavy functions like risk, compliance, and data governance are most often fully centralized. The downside: the central team becomes a bottleneck. Large enterprises that run every AI request through a central queue report nine-month average time to scale a pilot (C5 Insight 2025), compared to 90 days at mid-market firms with leaner structures.

Who does this: Smaller enterprises and regulated industries early in their AI journey. Capital One started centralized before evolving.

Decentralized / Federated

Each business unit runs its own AI program with minimal central coordination. This promotes speed and domain-specific innovation. The problems are predictable: duplicated infrastructure, inconsistent governance, incompatible tools, and shadow AI proliferation. BCG reports that 54% of employees would use unauthorized AI tools (BCG AI Radar, 2025) — decentralized models exacerbate this.

Who does this: Highly diversified conglomerates where business units have minimal overlap. Netflix operates a decentralized model with a centralized data platform (Metaflow).

Hub-and-Spoke (The Dominant Model)

A lean central hub sets enterprise standards for governance, evaluation frameworks, guardrails, MLOps, and cost controls. Business units operate as spokes, building and scaling use cases on shared infrastructure with local domain expertise. IBM’s CAIO survey (n=600+, 2025) finds this model delivers 36% higher ROI than decentralized alternatives.

Who does this: JPMorgan Chase (hub-and-spoke across its $18B technology budget), Walmart (centralized Element AI platform with 200+ distributed AI agents for store operations), Capital One (hub-and-spoke with MIT partnership and mandatory AI training for product managers).

McKinsey’s 2025 survey confirms the pattern: organizations partially centralize talent and adoption while fully centralizing governance — a textbook hub-and-spoke configuration. Companies below $500M in revenue are more likely to stay fully centralized, lacking the scale to justify distribution.

The Chief AI Officer: From Experiment to Mandate

The fastest-moving organizational change in AI governance is the emergence of the CAIO. IBM’s Institute for Business Value surveyed 600+ CAIOs across 22 geographies in Q1 2025 and found:

  • 26% of organizations now have a CAIO, up from 11% in 2023 — a 136% increase in two years
  • 57% of CAIOs were promoted internally, not hired externally — this is not a “hire a unicorn” play
  • 57% report directly to the CEO or Board, signaling that organizations treat AI as a strategic priority, not a technical function
  • 61% control their organization’s AI budget
  • 76% say other C-suite officers consult them on AI decisions
  • 66% of CAIOs expect most organizations will have one within two years

The reporting structure reveals how seriously organizations take the role:

Reports To Percentage
CEO 40%
CIO 24%
CTO 15%
CDO 10%
Other C-suite 11%

When the CAIO reports to the CEO, it signals AI as a business strategy. When the CAIO reports to the CIO or CTO, it signals AI as a technology initiative. The data shows the market is tilting toward the strategic framing — and organizations where CAIOs report to the CEO are overrepresented among top performers.

The CAIO role delineates clearly from existing C-suite positions: the CDO ensures data governance and quality (the “what”), the CTO manages platform and infrastructure (the “how”), and the CAIO defines AI investment strategy and risk (the “why” and “where”).

Board-Level Governance: More Form Than Substance

Board-level AI oversight is now a baseline expectation from institutional investors. An analysis of S&P 100 proxy disclosures by Harvard Law School Forum on Corporate Governance (March 2026) found:

  • 54% of S&P 100 companies disclose board-level AI oversight
  • 45% maintain disclosed AI policies
  • Only 28% disclose both oversight and a formal policy — the real governance bar
  • 63% of companies with oversight designate it to a specific committee (usually audit or technology), while 37% assign it to the full board
  • 44% of Fortune 100 companies now include AI competency in director qualification descriptions, up from 26% one year earlier (EY 2025)

The Sedgwick 2026 survey (n=300 Fortune 500 senior leaders) paints a sharper picture of the gap:

  • 70% have AI risk committees
  • 67% report progress on AI infrastructure
  • 41% have a dedicated AI governance team
  • 14% are fully ready for AI deployment

The 56-percentage-point gap between “we have a committee” (70%) and “we are ready” (14%) is the defining number. Governance structures are being built faster than operational capabilities, leaving most organizations with impressive org charts and mediocre execution.

Team Composition and Budget Reality

IBM’s CAIO research and practitioner analyses converge on budget ranges by maturity:

Growth stage (5-8 people, $1M-$2M annually): AI Product Manager, 2-3 ML Engineers, 1-2 Data Engineers, AI Architect, MLOps Engineer. This is adequate for proof-of-concept work and initial governance.

Enterprise scale (15-50+ people, $5M-$15M+ annually): Adds AI ethics/governance specialists, domain-specific AI leads embedded in business units, FinOps for AI cost governance, security engineers, and change management resources.

The broader budget picture: AI now accounts for roughly 12% of IT budgets in 2025, up from 10% months earlier. But the composition of AI spending is shifting. The Draup 2026 report on Fortune 500 hiring found that AI governance and model risk skill demand grew 81% year-over-year — the fastest-growing AI skill category. This outpaced growth in technical AI roles (roughly 30% YoY for individual contributors). The market is telling a clear story: companies have hired enough data scientists. What they lack are people who can govern, scale, and operationalize what those data scientists build.

Why Most AI CoEs Underperform

The failure rate for enterprise AI is stark. MIT’s State of AI in Business 2025 report found only 5% of AI solutions reach production. IBM’s 2025 data shows only 25% of AI initiatives have delivered expected ROI, and just 16% have scaled enterprise-wide. Informatica’s CDO Insights 2025 survey identifies the top obstacles: data quality and readiness (43%), lack of technical maturity (43%), and shortage of skills (35%).

AI Centers of Excellence fail for specific, predictable reasons:

They become approval bottlenecks. Central CoE teams that review every AI initiative create a queue. The enterprise adds 50 use cases a quarter; the CoE can process 10. Business units wait, then go around the CoE entirely — generating the shadow AI problem the CoE was created to prevent.

They focus on technology, not workflow redesign. Accenture’s 2025 data shows 3x more gen AI budgets target technology versus people. BCG’s 10-20-70 rule (10% algorithms, 20% technology, 70% people and processes) is cited widely and followed rarely.

They lack executive authority. A CoE without budget control or C-suite sponsorship is a committee, not a center of excellence. IBM’s data that 61% of CAIOs control the AI budget — and that CAIO-led organizations see 10% higher ROI — suggests budget authority is a prerequisite, not a reward for success.

They measure activity, not outcomes. The number of pilots launched, models trained, or tools deployed tells you nothing about business value. The 42% of companies that abandoned most of their AI initiatives in 2025 (up from 17% in 2024) were not suffering from a lack of activity.

Key Data Points

Metric Finding Source
Hub-and-spoke ROI advantage 36% higher ROI vs. decentralized IBM IBV 2025, n=600+ CAIOs
CAIO adoption rate 26% of organizations (up from 11% in 2023) IBM IBV Q1 2025, n=600+
Fortune 500 AI readiness Only 14% fully ready Sedgwick 2026, n=300
Fortune 500 with AI risk committees 70% Sedgwick 2026, n=300
S&P 100 disclosing board AI oversight + policy 28% Harvard Law Forum 2026
AI initiatives delivering expected ROI 25% IBM 2025
AI projects reaching production 5% MIT State of AI in Business 2025
Companies abandoning most AI initiatives 42% (up from 17% in 2024) Industry surveys 2025
AI governance skill demand growth 81% YoY Draup Fortune 500 hiring 2026
CAIO reporting to CEO 40% IBM IBV 2025
AI share of IT budgets 12% (2025) Industry benchmarks 2025

What This Means for Your Organization

The data points to a paradox. Most Fortune 500 companies now have the formal governance architecture for AI — committees, policies, even CAIOs. What they lack is the operational musculature to make that architecture produce results. The 56-point gap between “we have a risk committee” (70%) and “we are ready” (14%) is not a governance problem. It is an execution problem dressed in governance clothing.

The first decision is structural: hub-and-spoke outperforms alternatives by a measurable margin (36% higher ROI). If your AI efforts are either fully centralized and creating bottlenecks, or fully decentralized and creating chaos, the path is clear. A lean central team that owns governance, standards, and shared infrastructure — paired with business unit teams that own use case execution — is the configuration that scales.

The second decision is leadership. Organizations with CAIOs see higher ROI, but the reporting line matters. A CAIO who reports to the CIO inherits IT’s mandate: efficiency, security, stability. A CAIO who reports to the CEO inherits the business mandate: growth, differentiation, competitive position. The 40% of CAIOs reporting to CEOs are disproportionately at high-performing organizations. If your AI leader reports three levels below the CEO, you have already answered the question of whether AI is strategic or tactical.

The third decision is resource allocation. The market’s hiring patterns — 81% growth in governance skills versus 30% in technical AI roles — signal that the bottleneck has shifted. Most organizations do not need more data scientists. They need people who can govern AI usage, manage AI-related risk, measure business outcomes, and drive organizational change. Staffing your CoE with only technologists is building a sports car without a driver.

Sources

Created by Brandon Sneider | brandon@brandonsneider.com March 2026