Evaluating AI Model "Openness": A Procurement Framework for GCs and CTOs

See also (wiki): ai-vendor-contracts, ai-platform-selection, model-risk-management

Executive Summary

• Every major AI vendor markets their model as “open,” but the term is legally and practically meaningless without a structured framework. Forrester’s April 2026 AI Model Openness Framework (MOF) introduces three evaluation dimensions — Reproducibility, Usage Rights, and Community Momentum — that translate vendor claims into enterprise-grade procurement criteria.
• The Usage Rights dimension is where most enterprise deals break down. A model can score high on reproducibility (public weights, public code) and still carry field-of-use restrictions, MAU caps, or unilateral amendment rights that disqualify it from regulated production use. Marketing calls it “open.” Your GC calls it a legal review.
• Reproducibility maps directly to regulatory audit requirements. EU AI Act Article 13 high-risk system documentation, NIST AI RMF transparency requirements, and SOC 2 vendor assessments all require varying degrees of model documentation — documentation that only highly reproducible models provide.
• Community Momentum is the procurement dimension most commonly ignored. A model with no active maintainers, a single corporate sponsor, or a governance structure that allows license amendments (see: Llama) is a long-term operational risk, not just a legal one.
• The landscape is bifurcating. Google (Gemma 4) and Alibaba (Qwen3+) moved to genuine Apache 2.0 in 2025–2026. Meta (Llama) and DeepSeek retain custom licenses with enterprise-hostile clauses. The Forrester MOF gives procurement teams a repeatable method to score any model against these dimensions before the contract conversation starts.

Why “Open Source” Is Not a Procurement Specification

The AI model market has developed a binary vocabulary — open versus closed, open source versus proprietary — that does not map to enterprise procurement reality. The Open Source Initiative defines open source as permitting use “in any field of endeavor.” By that standard, the three most widely deployed “open” models in enterprise settings — Llama, pre-2026 Gemma, and DeepSeek — are not open source. They are source-available models with commercial restrictions.

This gap between marketing language and legal reality creates three downstream procurement failures:

1. Approved-vendor lists accept models that don’t qualify. Most enterprise approved-license lists cover Apache 2.0, MIT, and BSD. A Llama deployment that clears security review but not legal review ships anyway because no one checked the license against the list.

2. Regulated-use documentation gaps surface late. EU AI Act high-risk systems require training data documentation and algorithm transparency (Article 13). A model marketed as reproducible but with no training data disclosure fails this requirement at audit, not at procurement.

3. License amendments create post-deployment liability. Meta’s Llama Community License permits Meta to amend acceptable-use terms unilaterally. A model that was compliant at deployment can become non-compliant after a license update — with no notification obligation to existing users.

Forrester’s Model Openness Framework (MOF) provides a structured response to this problem. Rather than asking “is this model open source?” — a question vendors always answer affirmatively — it asks three questions with specific, scorable criteria.

The Three-Dimension Framework

Dimension 1: Reproducibility

Reproducibility assesses whether an enterprise can independently verify, recreate, or audit the model. The MOF evaluates:

• Availability of preprocessing, training, evaluation, and inference code
• Access to training data or verifiable source documentation
• Training methodology disclosure (algorithms, hyperparameters, evaluation benchmarks)
• Hardware and software environment specifications

Enterprise procurement application: Reproducibility directly determines regulatory compliance posture. NIST AI RMF requires transparency documentation. EU AI Act Article 13 (high-risk systems, enforcement August 2026) requires model documentation including training data sources and algorithmic logic. A model that fails Reproducibility cannot meet these requirements regardless of how its licensing is structured.

Note: This table reflects the publicly available documentation as of April 2026. “Audit-ready” means sufficient disclosure to meet EU AI Act Article 13 documentation requirements — not a legal opinion.

Dimension 2: Usage Rights

Usage Rights evaluates whether the model can actually be deployed in a production enterprise environment under the terms of its license. The MOF examines:

• Commercial use permissions and field-of-use restrictions
• MAU thresholds and affiliate aggregation clauses
• Documentation quality and cloud deployment options
• Vendor support and SLA availability

The enterprise failure pattern: A model can score high on Reproducibility and fail Usage Rights entirely. Llama’s Community License restricts use if your platform (and all affiliates with 50%+ ownership) exceeds 700 million monthly active users — an MAU aggregation clause that creates compliance exposure in M&A scenarios. DeepSeek’s license prohibits using model outputs to train competing models — a restriction that affects any enterprise doing fine-tuning or knowledge distillation.

Usage Rights is the dimension legal teams must own. Security teams approve the model. Legal teams approve the license. When these reviews run in parallel without a shared framework, deals close on technical approval and legal issues surface after deployment.

Dimension 3: Community Momentum

Community Momentum measures whether the model will still be maintained, updated, and governed three years from now. The MOF evaluates:

• Development activity and release cadence
• Bug response time and community responsiveness
• Contributor diversity (single-sponsor vs. multi-stakeholder)
• Governance structures and license amendment risk

Why this matters operationally: An enterprise that builds production workflows on a model with low Community Momentum faces two risks: (1) security vulnerabilities go unpatched as the maintainer community shrinks; (2) the corporate sponsor changes license terms without community recourse. The Llama license permits unilateral Meta amendments — a governance structure that gives Community Momentum a low score on the MOF regardless of current development velocity.

Models with Apache 2.0 licenses and multi-stakeholder contributor bases (Gemma 4, Qwen3+) score higher because the license cannot be revoked and development continuity is not dependent on a single corporate sponsor’s priorities.

Key Data Points

Source: Forrester AI Model Openness Framework (April 17, 2026), cross-referenced against primary license texts. See also: research/16-procurement-contracting/open-source-ai-license-exposure.md for detailed Llama clause analysis.

Temporal tier: TIER 1 — Forrester framework published April 17, 2026; license comparisons verified against current license texts.

What This Means for Your Organization

Three practical applications of the Forrester MOF for mid-market procurement:

1. Add the three dimensions to your model evaluation checklist before security review starts. Most organizations route AI model selection through security review (SOC 2, CAIQ) and technical evaluation. Legal review of Usage Rights happens late — often after the business unit has already committed. Running the MOF at the start of evaluation surfaces the Usage Rights questions before the deal has momentum.

2. Use Reproducibility score as a proxy for regulatory readiness. If any AI deployment touches EU data subjects (customers, employees, or vendors in the EU), EU AI Act Article 13 documentation requirements apply to high-risk systems from August 2026. A model with low Reproducibility — no training data disclosure, no methodology documentation — cannot meet these requirements at audit. The Forrester MOF’s Reproducibility dimension is a faster proxy for this gap than reading Article 13 directly.

3. License type predicts long-term operational risk. Apache 2.0 models (Gemma 4, Qwen3+) carry lower long-term license risk than custom-license models (Llama, DeepSeek) because the license cannot be amended retroactively. For workloads that will be in production for 3+ years, this distinction belongs in your build-vs-buy and vendor selection criteria alongside the technical evaluation.

The Forrester MOF’s Excel scoring template is client-gated, but the three-dimension structure is freely available and sufficient to build an internal procurement rubric. If you’re in the middle of a model selection decision and want to stress-test your evaluation criteria against these dimensions, I’d welcome the conversation — brandon@brandonsneider.com.

Sources

For detailed Llama clause analysis (700M MAU threshold, affiliate aggregation, unilateral amendment), see: research/16-procurement-contracting/open-source-ai-license-exposure.md.

Brandon Sneider | brandon@brandonsneider.com April 2026

See also (wiki)

• ai-vendor-contracts — procurement terms for AI model licensing, including field-of-use and MAU clause analysis
• model-risk-management — model reproducibility, audit documentation, and SR 11-7 alignment
• ai-platform-selection — platform vs. point-solution trade-off and build/buy/integrate decision framework