← Agent Frameworks 🕐 9 min read

Agent Frameworks

Static-Index Retrieval for Wiki and Markdown Sites

Brandon Sneider · June 2026

> **Source credibility: MEDIUM-HIGH.** Primary product docs are strong for implementation

Scope date: June 5, 2026.
Deployment target: Cloudflare Pages static site serving research/wiki Markdown and HTML.

See also (wiki): wiki/enterprise-rag-architecture.md - wiki/agentic-ai-governance.md - wiki/data-readiness.md

Source credibility: MEDIUM-HIGH. Primary product docs are strong for implementation details: Pagefind, llms.txt, Stork, SQLite WASM, DuckDB-Wasm, Cloudflare AI Search, Qdrant, Weaviate, Tantivy. Ecosystem adoption claims around llms.txt remain MEDIUM because it is a proposal and not a crawler-enforced web standard.

Executive Summary

Static knowledge sites should treat “agent retrieval” as a layered access problem, not as one search box. For a Cloudflare Pages site, the best practical architecture is:

Root discovery: /llms.txt plus sitemap links and per-page Markdown mirrors.
Browser/user retrieval: Pagefind generated at build time from HTML, with filters for category, type, and date.
Agent curl workflow: agents fetch /llms.txt, choose one or more Markdown document URLs, then fetch clean index.md pages directly instead of scraping HTML.
Optional static machine index: publish a compact JSONL manifest of titles, URLs, dates, categories, headings, and short snippets for CLI agents that want structured discovery without loading the Pagefind runtime.
Hybrid retrieval only when needed: BM25 + embeddings + reranking is valuable for large or ambiguous corpora, but it introduces either a Worker/API service or large client-side assets. Keep it out of the first static-site layer unless evaluation proves Pagefind + Markdown discovery is insufficient.

For this repo specifically, the live implementation is already close to the recommended baseline: site/package.json builds Pagefind, site/src/docs.njk marks article bodies with data-pagefind-body, site/src/llms.txt.njk emits a root Markdown index, and site/src/docs-md.njk emits one Markdown mirror per document.

Tooling Findings

llms.txt and llms-full.txt

Jeremy Howard’s September 3, 2024 proposal defines /llms.txt as a Markdown file that helps LLMs use a site at inference time, with brief guidance and links to detailed Markdown files. The proposal explicitly recommends clean Markdown page versions, and says the root file should include an H1, optional summary, guidance, and H2 sections containing file lists. It also says an Optional section can be skipped when shorter context is needed.

Practical reading: llms.txt is an agent entry point, not a ranking protocol. It coexists with sitemap and robots.txt, but should not be treated as a guarantee that major AI providers will crawl or cite the site. For a coding/research agent that can use curl, it is still high leverage because it removes discovery friction.

Recommended for State of AI:

Keep /llms.txt short enough to fetch cheaply.
Add a generated /llms-full.txt only if the full corpus stays within a sane transfer budget, or make it category-sharded: /research/19-agent-frameworks/llms-full.txt.
Preserve per-page Markdown URLs because they are more useful than HTML for agents.

Source: llms.txt proposal, Jeremy Howard, published September 3, 2024, accessed June 5, 2026: https://llmstxt.org/

Pagefind

Pagefind is the best default static search layer for this site. It works against static HTML output, generates a static search bundle after the site build, exposes a JavaScript API, and splits its index into chunks so the browser loads only a small subset of the index. The project states that a 10,000-page site can search with total network payload under 300 kB, including the library, with most sites closer to 100 kB.

Practical reading: Pagefind is designed for exactly the Cloudflare Pages case: no server, HTML generated by Eleventy, browser-side search, low bandwidth, filters, metadata, and section-level results. It is not a general machine API, but the generated index can support human search and agents operating through the browser.

Recommended for State of AI:

Keep Pagefind as the browser search default.
Continue using data-pagefind-body, data-pagefind-filter, and date sort metadata.
Add a small agent-facing endpoint only if agents need JSON search results without loading the browser UI; do not replace Pagefind for the user-facing search page.

Source: Pagefind documentation, version 1.5.2 page accessed June 5, 2026: https://pagefind.app/

MiniSearch, FlexSearch, and Lunr

MiniSearch, FlexSearch, and Lunr are useful when the corpus can fit in client memory as a single JSON/index payload. Lunr describes itself as a small browser full-text library that indexes JSON documents and avoids a server when the data already sits in the client. MiniSearch targets in-memory browser/Node full-text search with prefix, fuzzy search, ranking, field boosting, and filters, but similarly assumes the indexed data can fit locally.

Practical reading: these are better for small documentation apps or bespoke command palettes than for a growing research corpus. They are simpler than Pagefind if the corpus is tiny; they are worse once payload and memory become the limiting factors.

Recommended for State of AI: do not switch from Pagefind to a monolithic JS index. Use MiniSearch only for a narrow local feature such as searching a small command/help list.

Sources accessed June 5, 2026:

Lunr docs: https://lunrjs.com/docs/index.html
MiniSearch docs: https://lucaong.github.io/minisearch/index.html

Stork and Tantivy

Stork is a static-site search engine that builds a serialized compressed .st index from a TOML config. It can index plain text, subtitles, HTML, and Markdown, and its JS/WASM frontend loads the index in the browser. Its docs show static-site-generator integration by generating the config during Eleventy build and then running stork build.

The main caveat is WebAssembly/CSP: Stork’s self-hosting docs note that WASM execution can force script-src allowances such as unsafe-eval in some browsers, weakening a strict CSP. For a site that already has Pagefind, this is hard to justify unless Stork’s ranking or index-size profile beats Pagefind on a measured corpus.

Tantivy is a Rust Lucene-like full-text search library. It supports tokenizers, stemming, custom tokenizers, and precise field-level indexing control. It is a strong build-time or Worker-side search component, but not the simplest static Pages browser layer.

Recommended for State of AI: keep Tantivy/Stork as a second-track experiment only. If using Tantivy, put it behind a benchmark harness or Worker/CLI artifact, not the default static UI. If using Stork, measure CSP impact before deployment.

Sources accessed June 5, 2026:

Stork build docs: https://stork-search.net/docs/build
Stork self-hosting docs: https://stork-search.net/docs/self-hosting
Tantivy tokenizer docs: https://docs.rs/tantivy/latest/tantivy/tokenizer/index.html

SQLite, Parquet, and Static Structured Indexes

SQLite WASM enables SQLite in modern WASM-capable browsers. DuckDB-Wasm runs analytical SQL in the browser and can load data from Arrow, CSV, JSON, and remote Parquet URLs. This makes SQLite/DuckDB/Parquet attractive for structured metadata indexes: manifest rows, headings, entities, tags, citations, and source-file metadata.

Practical reading: these are excellent for faceted metadata and offline corpus analysis, but overkill for ordinary keyword search. Parquet is attractive for columnar metadata and agent analytics; SQLite FTS5 is attractive for local CLI workflows; neither should be the first browser search choice unless agents need SQL.

Recommended for State of AI:

Publish search-manifest.jsonl or docs-index.json first.
Consider docs-index.parquet only if the site grows into analytical workflows, e.g. “find all benchmark notes sourced after May 2026 that cite arXiv and mention reranking.”
Keep any SQLite/Parquet artifact append-only and build-generated so Cloudflare Pages can serve it as a static asset.

Sources accessed June 5, 2026:

SQLite WASM docs: https://www.sqlite.org/wasm
DuckDB-Wasm blog/docs: https://duckdb.org/2021/10/29/duckdb-wasm

BM25 + Embedding Hybrid Retrieval

Hybrid retrieval is now a mainstream RAG architecture: Cloudflare AI Search documents hybrid search as vector and keyword search in parallel with result fusion, raw BM25 keyword scores, vector scores, RRF/max fusion, and optional cross-encoder reranking. Qdrant likewise supports dense+sparse prefetch with RRF. Weaviate’s docs describe hybrid as BM25 plus vector search with weighting, useful when either keyword or vector search alone is weak.

Practical reading: hybrid retrieval is right for production RAG, but not necessarily for a pure static site. It requires embeddings, model/version management, score fusion, and usually a server/API layer. Cloudflare AI Search may be the lowest-friction Cloudflare-native path if the site later needs managed hybrid retrieval, but it is no longer “just static Pages.”

Recommended for State of AI:

Use Pagefind and Markdown mirrors as the static baseline.
Add an evaluation harness before adding embeddings: sample 50 agent questions, record whether /llms.txt + Markdown + Pagefind finds the right page in top 5.
If top-5 page recall is inadequate, test Cloudflare AI Search hybrid retrieval or an external Qdrant/Weaviate pipeline against the same query set.
Prefer reranking only after first-stage retrieval is measured; cross-encoder reranking improves precision but adds latency/cost and operational state.

Sources accessed June 5, 2026:

Cloudflare AI Search hybrid docs: https://developers.cloudflare.com/ai-search/configuration/indexing/hybrid-search/
Qdrant hybrid query docs: https://qdrant.tech/documentation/search/hybrid-queries/
Weaviate hybrid search docs: https://docs.weaviate.io/academy/py/zero_to_mvp/queries_2/hybrid

Context Compression And Static Retrieval

ScaleDown’s official documentation is a useful adjacent signal for agent-facing static retrieval. It frames task-specific SLMs as a context-extraction layer that keeps only information relevant to a prompt, and its Python docs describe domain-keyword preservation, AST/BM25 code-context pruning, local embeddings with FAISS, and span extraction with offsets and confidence scores.

Practical reading: compression is not a replacement for retrieval quality. For wiki and finance workflows, the dangerous failure mode is removing the small exact evidence that makes an answer verifiable: dates, source names, tickers, table headers, formulas, citations, caveats, and authority boundaries. Compression belongs after source discovery and before answer generation, with explicit tests for evidence preservation.

Recommended for State of AI:

Keep /llms.txt, Markdown mirrors, and Pagefind as the retrieval baseline.
Add compression only as an evaluated module, not as an unconditional build step.
Score compressed-context outputs for preservation of publish dates, retrieval dates, source URLs, section headings, financial identifiers, formulas, and caveats.
Do not infer ScaleDown runtime compatibility with MLX, GGUF, CUDA, ROCm, SageMaker, Bedrock, or AWS Neuron from the docs unless a separate source proves that backend.

Source accessed June 6, 2026:

ScaleDown documentation: https://docs.scaledown.ai/Documentation

Recommended Static Architecture

Layer 1: Agent Discovery

Expose these stable root or category paths:

/llms.txt
/sitemap.xml
/research/<category>/
/research/<category>/<slug>/index.md
/technical/<category>/<slug>/index.md

Agent workflow:

curl -L https://stateofai.pages.dev/llms.txt
curl -L https://stateofai.pages.dev/research/19-agent-frameworks/<slug>/index.md

Expected behavior: agents should start from /llms.txt, use category labels and titles to select candidate pages, then fetch Markdown mirrors directly. This is robust, cheap, and does not depend on JavaScript.

Layer 2: Browser Search

Continue using Pagefind:

eleventy && pagefind --site _site

Keep category/type/date metadata in the HTML so users and browser agents can filter results. This is already present in the repo.

Layer 3: Agent Manifest

Add a generated static manifest if agent retrieval needs a structured index:

{"title":"Re-Ranking Architectures for Enterprise RAG","url":"/research/19-agent-frameworks/retrieval-reranking-enterprise-rag/","markdown_url":"/research/19-agent-frameworks/retrieval-reranking-enterprise-rag/index.md","category":"Agent Frameworks & Production Patterns","date":"2026-05-01","headings":["Executive Summary","Why Embedding Similarity Alone Fails"],"summary":"..."}

Prefer JSONL for easy streaming with curl, jq, and shell tools:

curl -L https://stateofai.pages.dev/docs-index.jsonl \
  | jq -r 'select(.category | test("Agent Frameworks")) | [.title,.markdown_url] | @tsv'

Layer 4: Optional Static Context Bundles

Generate category-scoped context files:

/research/19-agent-frameworks/llms-full.txt
/research/21-benchmarks/llms-full.txt

Do not make a single unbounded whole-site llms-full.txt unless its size is monitored. Large concatenated files become expensive and less useful than sharded bundles plus per-page Markdown.

Layer 5: Optional Managed Hybrid Retrieval

If measured static retrieval quality is poor:

Create a benchmark query set from actual agent/user questions.
Score Pagefind top-5, Markdown discovery top-5, and hybrid retrieval top-5.
Promote Cloudflare AI Search, Qdrant, or Weaviate only if hybrid retrieval materially improves recall or reduces time-to-correct-page.

Decision Matrix

Approach	Static Pages fit	Agent fit	Best use	Main risk
`/llms.txt` + per-page Markdown	High	High	Agent discovery and clean fetch	Not universally honored by crawlers
`/llms-full.txt`	Medium	Medium-High	Small or sharded context packs	File grows beyond useful context/transfer budget
Pagefind	High	Medium	Human/browser search	JS API, not a simple curl API
MiniSearch/FlexSearch/Lunr	Medium	Low-Medium	Small in-memory indexes	Monolithic payload and memory growth
Stork	Medium	Medium	WASM static search with compressed index	CSP and WASM deployment overhead
Tantivy	Low-Medium	Medium	Build-time/Worker-side BM25 experiments	More infrastructure than Pagefind
SQLite/Parquet static manifest	Medium	High	Structured metadata and CLI analytics	Overkill for plain keyword search
BM25 + embeddings + reranker	Low static / High managed	High	Production RAG over large corpus	Requires service, model ops, eval harness

Bottom Line

For a Cloudflare Pages wiki/research site, the efficient retrieval path for AI agents is not “ship a vector database in the browser.” It is:

Clean Markdown mirrors.
A curated /llms.txt.
Pagefind for human/browser search.
A small JSONL manifest for CLI agents.
Measured hybrid retrieval only after static retrieval misses real queries.

This keeps the site static, cheap, cacheable, and easy for agents to operate with curl, while leaving a clear upgrade path to Cloudflare AI Search or another hybrid system if the corpus outgrows lexical/static discovery.