← Agent Frameworks 🕐 14 min read
Agent Frameworks

NVIDIA Enterprise RAG Blueprint — Architecture and Enterprise Decision Guide

Brandon Sneider · May 2026

> **Source credibility:** NVIDIA TIER 2 — vendor-produced technical documentation and open-source reference implementation (Apache 2.0).

See also (wiki): wiki/enterprise-rag-architecture.md · wiki/agentic-ai-governance.md · wiki/data-readiness.md · wiki/ai-cybersecurity.md · wiki/hitl-deployment-pattern.md · wiki/model-risk-management.md · wiki/vendor-security-questionnaires.md · wiki/mcp-security-governance.md

Source credibility: NVIDIA TIER 2 — vendor-produced technical documentation and open-source reference implementation (Apache 2.0). Architecture claims are reproducible via the public GitHub repo (NVIDIA-AI-Blueprints/rag). Component-level performance claims (15× extraction speed, GPU-accelerated indexing) are from NVIDIA engineering blogs, not independent benchmarks. No n= survey data. Hardware requirements and NIM model specs are verifiable against NVIDIA support matrices. These case studies are vendor-published and represent selected wins with no control group and no independent verification. Cross-reference against: METR RCT (experienced developers 19% slower), CMU study (40.7% code complexity increase), Atlan 200-deployment analysis (median +159.8% ROI requires workflow redesign first).

Executive Summary

The NVIDIA RAG Blueprint is a production reference implementation, not a product. It is a composed stack of NVIDIA NIM microservices, a LangChain-based orchestration layer, and a pluggable vector database, delivered as open-source code with Docker Compose and Helm deployment paths. The primary audience is enterprise AI platform teams that want GPU-accelerated RAG with multimodal ingestion and do not want to build the plumbing themselves.

Key findings for enterprise technology leaders:

  • The blueprint solves the extraction problem first. Most enterprise RAG implementations fail at ingestion — they cannot reliably extract structured content from PDFs, tables, charts, or infographics. NVIDIA addresses this with six dedicated NIM microservices for extraction before a single embedding is computed.
  • The architecture is explicitly designed to be swapped. Every major component — vector database, embedding model, reranking model, LLM, guardrails — is an environment variable away from replacement. This is materially different from vertically integrated RAG SaaS platforms.
  • Hardware requirements are non-trivial. Self-hosted deployment requires 2× H100 or 2× B200 at minimum for Docker, 8× H100 or equivalent for Kubernetes. Guardrails add 2 additional GPUs. This is infrastructure investment, not a software subscription.
  • The “NVIDIA-hosted endpoints” option (pointing NIM calls to integrate.api.nvidia.com) reduces local GPU requirements to near-zero but introduces API cost and data egress for every query and every ingested document.
  • Governance is opt-in, not default. Guardrails (content safety + topic control) are ENABLE_GUARDRAILS=false by default. Self-reflection is off by default. Reranking is on. Every quality-improving feature involves a latency and GPU cost tradeoff.

This file covers the NVIDIA blueprint’s specific architecture, deployment constraints, and enterprise decision criteria. For the re-ranking architecture that underpins the retrieval pipeline, see retrieval-reranking-enterprise-rag.md in this directory — this file does not duplicate that coverage.

What Problem This Blueprint Solves

Enterprise RAG deployments fail at two distinct layers:

Layer 1 — Extraction. Enterprise documents are not text files. They are PDFs with tables spanning multiple pages, slide decks with charts as images, technical manuals with infographics, audio transcripts, and scanned documents with embedded text. Naive text extraction produces garbage that embeds as garbage. The NVIDIA blueprint’s primary architectural bet is that extraction quality determines RAG quality, and extraction requires specialized vision models, not generic PDF parsers.

Layer 2 — Retrieval precision. Even with clean text, embedding similarity alone does not produce task-relevant results. The blueprint defaults to dense embedding + reranking (dense-only search, reranker enabled), with hybrid search (BM25 + dense) as an opt-in configuration.

The blueprint does not claim to solve governance, model risk management, or enterprise change management. Those are out of scope by design.

Full Component Stack

Tier 1 — Extraction NIMs (ingestion pipeline)

Six NIM microservices process documents before any embedding:

NIM Function Default
NeMo Retriever Page Elements v3 Detects and classifies page layout elements (text blocks, figures, tables, headers) On
NeMo Retriever Table Structure v1 Extracts table structure — rows, columns, cell relationships — from visual table representations On
NeMo Retriever Graphic Elements v1 Identifies charts and infographics, classifies chart type On
NeMo Retriever OCR Optical character recognition for scanned documents and text-in-images On
NeMo Retriever Parse NIM Alternative to pdfium; full structural understanding of complex PDF layouts Optional
PaddleOCR NIM Legacy OCR option (baidu/paddleocr); still supported, NeMo Retriever OCR is now default Optional

Extraction modes by document type:

  • Text documents: pdfium (default) or Nemotron Parse (optional, GPU-intensive)
  • Tables: NeMo Table Structure NIM (enabled by APP_NVINGEST_EXTRACTTABLES=true, default on)
  • Charts/infographics: Graphic Elements NIM (charts on by default; infographics APP_NVINGEST_EXTRACTINFOGRAPHICS=false)
  • Images: VLM image captioning (off by default; requires separate VLM NIM)
  • Audio: Whisper-based segmentation (APP_NVINGEST_SEGMENTAUDIO=false by default)

Chunking defaults: 512-token chunks, 150-token overlap. Chunk size is tunable. PDF split processing (parallel page-level chunking for large documents) is off by default.

NVIDIA engineering claims 15× faster multimodal PDF extraction versus unaccelerated alternatives (NVIDIA developer blog, 2025). This claim reflects GPU-accelerated object detection for table/chart identification and should be treated as directional.

Tier 2 — Embedding and Reranking NIMs

NIM Model Function
llama-3_2-nv-embedqa-1b-v2 1B Llama-based Dense embedding for both document ingestion and query encoding
llama-3_2-nv-rerankqa-1b-v2 1B Llama-based cross-encoder Re-scores retrieved chunks by relevance; runs after first-stage retrieval
llama-3_2-nemoretriever-1b-vlm-embed-v1 1B VLM Multimodal embedding (image + text); Early Access as of this writing

The embedding and reranking models are small by design — 1B parameters each. This is a deliberate performance choice: keep the hot retrieval path fast, put inference budget into the generation model. The cross-encoder reranker runs at the default VDB top-k of 100 documents, returning top 10 after reranking.

Reranker threshold configuration:

  • RERANKER_SCORE_THRESHOLD: default 0.0 (no filtering). NVIDIA recommends 0.3–0.5 for production.
  • RERANKER_TOP_K: default 10 after reranking from VDB top-k of 100.

This is the same two-stage retrieval → reranking architecture analyzed in retrieval-reranking-enterprise-rag.md. The NVIDIA implementation uses a cross-encoder reranker (higher accuracy than ColBERT, higher latency than SPLADE). For the latency/accuracy tradeoff analysis, see that file.

Tier 3 — Vector Database

Milvus (default) with GPU-accelerated indexing via NVIDIA cuVS. Object storage via minIO for raw document chunks. Elasticsearch is supported as an alternative.

Milvus with cuVS provides GPU-accelerated ANN (approximate nearest neighbor) index construction and search. This matters at scale: for corpora above ~1M documents, GPU-accelerated indexing reduces index build time materially. For smaller corpora (<100k documents), the GPU overhead of Milvus may not justify the complexity versus a simpler in-memory vector store.

Hybrid search (BM25 + dense) is available: APP_VECTORSTORE_SEARCHTYPE=hybrid. Default is dense. NVIDIA’s accuracy benchmarks show hybrid search improving results for domain-specific content where keyword signals remain important alongside semantic similarity.

Multi-collection retrieval: The blueprint supports querying across multiple Milvus collections in a single request. This enables multi-tenant document isolation (each department keeps its knowledge base separate) while allowing cross-collection queries when needed.

Tier 4 — Orchestration Layer

RAG Orchestrator Server: LangChain-based. Manages the multi-turn conversation state, routes queries through the pipeline stages (query rewriting → retrieval → reranking → generation), and exposes OpenAI-compatible APIs. The OpenAI API compatibility means existing integrations targeting GPT models can point at the RAG server without client-side changes.

Pipeline stages and their latency contributions (from Prometheus/Zipkin observability):

  • retrieval_time_ms — vector search
  • context_reranker_time_ms — reranker scoring
  • llm_ttft_ms — time to first token from generation model
  • llm_generation_time_ms — full generation time
  • rag_ttft_ms — end-to-end TTFT including retrieval and reranking

Optional pipeline stages:

  • Query rewriting (ENABLE_QUERYREWRITER=false): LLM call to reformulate multi-turn follow-up questions into standalone queries. Adds latency.
  • Query decomposition: Breaks complex questions into sub-queries. Useful for multi-hop retrieval.
  • Self-reflection (ENABLE_REFLECTION=false): LLM judge evaluates retrieved context relevance (score 0–2) and response groundedness (score 0–2), loops up to MAX_REFLECTION_LOOP=3 times. Requires 8× A100/H100 for on-prem. Adds significant latency.
  • Dynamic filter expression creation: LLM generates Milvus metadata filters at query time.

Tier 5 — Generation NIM

Default: nvidia/llama-3.3-nemotron-super-49b-v1.5 — NVIDIA’s flagship reasoning-capable model, optimized for instruction following.

Optional reasoning mode: LLM_ENABLE_THINKING=true activates extended reasoning for Nemotron 3 models. Improves response quality; increases latency and token costs.

VLM generation: ENABLE_VLM_INFERENCE=true routes retrieved image chunks through a Vision Language Model (default: Nemotron-3-nano-30b-a3b-omni-reasoning) rather than text-only generation. Supports up to 4 images per citation. Required for use cases where the answer is encoded in a chart or diagram, not text.

Tier 6 — Guardrails NIMs (optional)

Two 8B models deployed as separate NIMs, each requiring a dedicated GPU with 48GB VRAM:

NIM Function
Llama 3.1 NemoGuard 8B Content Safety Multilingual detection of unsafe interactions (both input and output); triggers simple refusal responses
Llama 3.1 NemoGuard 8B Topic Control Constrains conversations to defined topic scope; prevents off-topic responses

Current limitations (as of this writing):

  • Not supported on B200 GPUs (H100 and A100 only)
  • Helm deployment not supported for guardrails
  • Jailbreak detection model not yet available out-of-the-box
  • Content safety and topic control models are single-turn trained — multi-turn conversation safety is inconsistent
  • Refusal responses are simple (“I’m sorry. I can’t respond to that.”) — no graceful degradation

Hardware cost of guardrails: 2 additional dedicated GPUs (2× H100 48GB or 2× A100 48GB) on top of the base deployment. This is a material infrastructure cost that security teams should weigh against the actual risk profile of the deployment.

For enterprise governance requirements mapped to OWASP and NIST frameworks, see cisco-ai-defense-token-audit.md and jpmc-lethal-trifecta-agentic-ai-security.md.

The “Decomposable and Configurable” Architecture Philosophy

NVIDIA explicitly frames this as a starting point, not a finished product. “Decomposable and configurable” means:

Every major component is a NIM endpoint URL. Swap the LLM by changing APP_LLM_MODELNAME. Point the embedding NIM at any OpenAI-compatible embeddings endpoint. Replace Milvus with Elasticsearch by changing APP_VECTORSTORE_NAME. No code changes required for model swaps.

Feature flags control the pipeline graph. Reranking, query rewriting, self-reflection, guardrails, VLM generation, hybrid search — each is an environment variable. Teams can disable every optional feature and profile them independently to understand their latency and accuracy contribution.

Library mode exists. In addition to the Docker microservice deployment, the pipeline can run as a Python library (pip install nvidia-rag). This enables integration into existing orchestration systems without deploying the full RAG server. Useful for teams already running LangChain or LlamaIndex pipelines who want NVIDIA’s extraction and retrieval quality.

Containerless lite mode. For evaluation or small deployments: no GPU required locally, NVIDIA-hosted NIM endpoints handle inference. Milvus Lite replaces the full Milvus cluster. Viable for proof-of-concept but not production at scale.

What cannot be swapped: The orchestration layer is LangChain-based. Teams with strong LlamaIndex or custom orchestration preferences will find this an integration point, not a strength. The extraction pipeline (NeMo Retriever NIMs) requires NVIDIA NGC access and NVIDIA GPU hardware — there is no CPU-only extraction path for production use.

Deployment Options and What Each Requires

Option 1: Docker Compose — Self-Hosted Models

The recommended path for enterprises with on-prem GPU clusters.

Minimum hardware:

  • 2× H100, or 2× B200, or 3× A100 SXM, or 2× RTX PRO 6000
  • 200GB free disk space (100–150GB for model downloads + containers + vector data)
  • Ubuntu 22.04, GPU Driver 560+, CUDA 12.9+

Additional if guardrails enabled: 2× dedicated H100 or A100 with 48GB VRAM each.

Startup sequence (order-dependent):

  1. NIMs (nims.yaml) — wait for health: LLM, embedding, reranking NIMs
  2. Vector database (vectordb.yaml) — Milvus + minIO
  3. Ingestor server (docker-compose-ingestor-server.yaml)
  4. RAG server (docker-compose-rag-server.yaml)
  5. Optional: Guardrails (docker-compose-nemo-guardrails.yaml)

Data sovereignty: all data stays on-prem. No external API calls unless NIM_ENDPOINT_URL is pointed at integrate.api.nvidia.com.

Option 2: Docker Compose — NVIDIA-Hosted Endpoints

For teams that want to evaluate the full stack without GPU infrastructure.

GPU requirement: 1× L40 GPU recommended for Milvus with GPU-accelerated indexing. Otherwise near-zero local GPU.

All NIM calls route to integrate.api.nvidia.com. Every document chunk and every query leaves the enterprise network. Not appropriate for confidential enterprise data. Acceptable for proof-of-concept with synthetic or public data.

Cost model: per-token API costs for every query (embedding + reranking + generation) and every ingested document (embedding).

Option 3: Kubernetes (Helm)

For production multi-tenant deployments at scale.

Minimum hardware: 8× H100-80GB, or 8× B200, or 9× A100-80GB SXM.

Supports NIM Operator for lifecycle management of NIM deployments. MIG (Multi-Instance GPU) support available for A100/H100 to partition GPUs for embedding and reranking NIMs separately.

Current limitations: NeMo Guardrails not supported in Helm deployment. Kubernetes is the path for scale but loses the guardrails capability until NVIDIA ships Helm support.

Option 4: AI Workbench

NVIDIA AI Workbench integration for developer laptops and workstations with consumer GPU hardware. Useful for individual developer evaluation. Not a production path.

Option 5: Python Library (Containerless)

pip install nvidia-rag

Embeds the pipeline logic into the calling application. Uses Milvus Lite for vector storage and NVIDIA-hosted endpoints for inference. No Docker, no containers, no GPU required. Useful for rapid prototyping and integration testing.

Latency and Scalability Characteristics

Observability built in: Prometheus metrics, Zipkin distributed tracing, Grafana dashboards. Every pipeline stage exports named metrics. This is materially better than most open-source RAG implementations where latency attribution requires custom instrumentation.

Key latency drivers (from pipeline metrics):

  • Vector search latency scales with corpus size and VDB top-k setting (default 100)
  • Reranker latency scales linearly with number of chunks reranked
  • Enabling query rewriting adds one LLM call before retrieval
  • Self-reflection adds up to 3× LLM calls after generation (each iteration checks context relevance + response groundedness, loops if below threshold)
  • Guardrails add 2 LLM calls (content safety model at input, topic control at output) with dedicated GPU serving

Ingestion throughput tuning:

  • NV_INGEST_CONCURRENT_BATCHES (default 4) × NV_INGEST_FILES_PER_BATCH (default 16) = 64 files in flight concurrently
  • Rule: CONCURRENT_BATCHES × FILES_PER_BATCH ≈ MAX_INGEST_PROCESS_WORKERS
  • PDF split processing (APP_NVINGEST_ENABLE_PDF_SPLIT_PROCESSING=false) enables parallel page-level processing for large PDFs — useful when ingesting a small number of very large documents

Benchmarking tooling: GenAI Perf (NVIDIA’s open-source tool, built on Triton perf_analyzer) measures TTFT, inter-token latency, and throughput across concurrency levels. RAGAS framework for RAG-specific accuracy evaluation (context recall, faithfulness, answer relevance).

Governance Requirements Built In

The blueprint includes several governance-relevant features that enterprise risk teams should inventory:

Content safety (opt-in): NemoGuard 8B Content Safety NIM provides input/output filtering for unsafe content. Covers multilingual content. Not enabled by default.

Topic control (opt-in): NemoGuard 8B Topic Control NIM restricts conversations to defined topic domains. Prevents the system from answering questions outside its intended scope.

Citations and transparency: The default UI returns citations with retrieved document references. API responses include the source documents that grounded each answer. This supports audit trails for regulated environments.

Telemetry and observability: Prometheus metrics + Zipkin traces provide a full audit trail of what was retrieved for which query. This is the data needed for model risk management reviews.

What is NOT built in:

  • Role-based access control on collections (no native RBAC — implement at the API gateway layer)
  • PII detection in ingested documents or generated responses
  • Differential privacy for the vector database
  • Watermarking of generated content
  • HITL checkpoints (the blueprint is fully automated; HITL must be inserted at the application layer)

For the HITL deployment pattern analysis, see enterprise-agentic-ai-operations-playbook.md.

How This Compares to What’s in retrieval-reranking-enterprise-rag.md

retrieval-reranking-enterprise-rag.md covers the architectural theory and benchmark data for re-ranking: why cross-encoders outperform ColBERT on precision, BEIR nDCG@10 numbers, latency tables, and the conditions under which reranking is not worth the cost.

This file covers the NVIDIA blueprint specifically: what it deploys, what it costs in hardware, what can be swapped, and when to use it. The two files are complementary.

Specific deltas this file adds:

  • The extraction layer (six NIM microservices for tables, charts, OCR, infographics, audio) — not covered in the reranking file
  • GPU hardware requirements for each deployment option
  • The guardrails add-on architecture and its GPU costs
  • The self-reflection loop (up to 3× iterative quality improvement)
  • VLM generation mode for image-heavy document corpora
  • The library and containerless deployment modes
  • Chunking and ingestion throughput configuration

Enterprise Decision Criteria: When to Use This vs. Build Custom

Use the NVIDIA blueprint when:

  • The enterprise already has or is acquiring H100/A100 cluster capacity and wants to maximize utilization
  • Documents to be ingested contain significant non-text content (tables, charts, infographics, scanned PDFs)
  • The team wants a proven component stack with minimal integration work and clear upgrade paths
  • OpenAI API compatibility is required for existing integrations
  • GPU-accelerated vector search at scale (>1M documents) is a requirement
  • The team wants Kubernetes Helm deployment with NIM Operator lifecycle management

Build custom (or use an alternative) when:

  • All data is clean text; the extraction layer adds GPU cost without extracting value
  • The enterprise is deeply invested in a specific orchestration framework (LlamaIndex, custom) that conflicts with LangChain
  • Confidential data cannot transit NVIDIA NGC for model download authentication
  • The deployment environment is CPU-only or cloud-native with no GPU access
  • The team requires Helm-based guardrails (not yet supported)
  • The budget cannot justify H100/B200 infrastructure for a RAG use case that could run on smaller hardware

The honest constraint: This is the right stack for enterprises that are building a GPU-accelerated AI platform where RAG is one of several workloads. It is over-engineered for a single department deploying a Q&A bot on a document corpus of fewer than 100k documents. The minimum viable version of this problem can be solved with a managed vector database (Pinecone, Weaviate Cloud) and a hosted LLM API at a fraction of the infrastructure investment.

Key Configuration Reference

Variable Default Effect
APP_NVINGEST_CHUNKSIZE 512 Token chunk size for ingestion
APP_NVINGEST_CHUNKOVERLAP 150 Overlap between adjacent chunks
APP_NVINGEST_EXTRACTTABLES true Table structure extraction via NIM
APP_NVINGEST_EXTRACTCHARTS true Chart extraction via NIM
APP_NVINGEST_EXTRACTINFOGRAPHICS false Infographic extraction (increases ingestion time)
APP_NVINGEST_EXTRACTIMAGES false VLM image captioning during ingestion
APP_NVINGEST_SEGMENTAUDIO false Audio segmentation and ingestion
APP_NVINGEST_PDFEXTRACTMETHOD pdfium nemotron_parse for complex PDFs (GPU-intensive)
APP_VECTORSTORE_SEARCHTYPE dense hybrid enables BM25 + dense retrieval
ENABLE_RERANKER true Cross-encoder reranking after first-stage retrieval
RERANKER_SCORE_THRESHOLD 0.0 Recommend 0.3–0.5 in production
ENABLE_GUARDRAILS false NemoGuard content safety + topic control
ENABLE_QUERYREWRITER false LLM-based query reformulation for multi-turn
ENABLE_REFLECTION false Iterative quality improvement via LLM judge
ENABLE_VLM_INFERENCE false VLM generation for image-heavy document corpora
LLM_ENABLE_THINKING false Extended reasoning mode for Nemotron 3 models
NV_INGEST_CONCURRENT_BATCHES 4 Parallel ingestion batch streams
NV_INGEST_FILES_PER_BATCH 16 Files per ingestion batch

Vendor Security Questionnaire Implications

For enterprises running security reviews on this stack (see vendor-security-questionnaires.md):

  • Data egress: Confirm whether NIM_ENDPOINT_URL points to on-prem or integrate.api.nvidia.com. If cloud, every query and document chunk leaves the network.
  • Model provenance: All NIMs are Llama-based (Llama 3.1, 3.2, 3.3 community licenses). License terms govern commercial use.
  • Authentication: NVIDIA NGC API key required for image pulls. Rotate on a schedule.
  • Network exposure: RAG server exposes port 8081, ingestor port 8082. Guardrails on separate ports. In production, proxy behind an API gateway with authentication.
  • Logging: Zipkin traces include traceloop.entity.input and traceloop.entity.output — query text and retrieved document snippets are in the trace logs. Ensure log retention policies are consistent with data classification.

Sources

  • NVIDIA-AI-Blueprints/rag GitHub repository (Apache 2.0), accessed May 2026
  • NVIDIA NeMo Retriever documentation, docs.nvidia.com/nemo/retriever
  • NVIDIA developer blog: “NVIDIA NeMo Retriever Delivers Accurate Multimodal PDF Data Extraction 15x Faster” (2025)
  • NVIDIA developer blog: “Finding the Best Chunking Strategy for Accurate AI Responses” (2025)
  • NVIDIA NIM support matrices for individual model components