Search blog posts, product pages, and docs by title, topic, or keyword. Try “RAG”, “compliance”, or “CI/CD”.
The Open Web Vector Initiative
We are building the other thing: a public, per-site retrieval index. Every site gets its own vector database, its own embeddings, and a chat endpoint grounded in its own words, with citations back to the page. Nothing is trained on. The corpus stays attributable to the site it came from — and any site owner can claim theirs.
Measured, not projected — read live from the public directory API. Measured 18 August 2026.
52% of the index is .gov, .edu and .org. That is not a filter we applied for the look of it — it is what happens when you follow citations and documentation links rather than commercial traffic.
diff.wikimedia.org, ipac.caltech.edu and blog.mozilla.org run past 7,000 pages each.
exoplanetarchive.ipac.caltech.edu — 929 pages, and 176 MB of extracted text: the densest corpus in the index.
A few hundred pages and a few thousand chunks. The index is wide rather than deep — the point is coverage of the open web, not a handful of giants.
A host that refuses stops at step 01 and costs us zero browser-seconds. Nothing is fetched, so there is nothing to delete later.
A site's content is never blended with another's, so retrieval cannot cite the wrong source — and a site owner can be handed exactly their own data, with nothing of anyone else's mixed in.
The pipeline discovers its next targets from the outbound links of the corpus it just built. Crawl, chunk, embed and serve all run on Cloudflare — Workers, Workflows, Browser Rendering and Workers AI — with Turso as the vector store.
The part that makes this different
Asking permission after you have taken the content is not asking permission. So the gate is the first thing in the pipeline, and a refusal costs us nothing to honour. This is engineering, not a values statement bolted on afterwards.
A site declaring search=yes, ai-train=no, ai-input=yes has permitted precisely what we do and forbidden precisely what we don't. We honour that distinction instead of collapsing every AI signal into a blanket no — so sites that block training crawlers are still indexed for retrieval. That is the point.
If robots.txt answers 403, 405 or 429, we do not treat "no restriction found" as "no restriction exists". Those hosts are retried, and then dropped — never quietly published on the assumption that silence meant yes.
Every fetch carries crawlPurposes: ["search", "ai-input"], so publishers can see what we are doing in their own Cloudflare analytics rather than having to take our word for it here.
6 of 12
In a screened sample of candidate hosts, half refused, and we listened. One of them blocks cloudflarebrowserrenderingcrawler by name — the exact renderer we use — so we skipped it. That is the honest hit rate, and a project that only reports the sites it did index is not reporting the interesting half.
The output belongs in public, so the work compounds for people who aren't us. But a Hugging Face dataset of raw page text is, in practice, a training corpus — that is the single most common use of the format. Publishing one would take content from sites that explicitly set ai-train=no and hand it to exactly the use they refused. So this ships in stages, and the stages are the whole point.
One row per site: host, page and chunk counts, corpus bytes, crawl date, embedding model, the site's declared AI signals, and our gate verdict. No content. Nobody has published a measured, per-host map of who permits what — that alone is worth having.
The vectors, plus source URL and offsets, with no reconstructable page content. Useful for retrieval research and embedding benchmarks; not a training corpus.
Restricted to hosts carrying no training reservation, plus public-domain material — with an explicit way for a site owner to say yes, which no comparable dataset offers. It waits on one piece of engineering: the gate verdict has to be stored durably per host alongside the crawl before any text-bearing release can be filtered correctly. That work comes first.
Every indexed host has a working chat endpoint that answers out of that host's own pages, and cites the page it answered from.
Open the directory to search, filter and export the full catalogue, or chat with any site in it.
If your site is already in the index, you can claim it by DNS or file verification and take over its assistant. If it isn't, you can put it there yourself — and if you'd rather we never did, the gate is already reading your robots.txt.
