# Case study: Supplements evidence RAG · answers the research actually supports

Grounded Q&A over PubMed dietary-supplement research — "does vitamin C help colds?" gets a cited answer from real studies, or a designed abstention. Stdlib-Python pipeline core; Cloudflare Workers AI + Vectorize on the dense side; deployed as a Cloudflare Python Worker at luisosio.dev/sr/v1/. Fully open source; the repo and the eval report ship together.

**Evidence:** Live — pipeline built, measured, and deployed (Python Worker, D1 corpus + BM25 postings, Vectorize dense leg). **Case page:** https://luisosio.dev/work/supplements-rag/

## Problem

Everyone has asked "does this supplement actually do anything?" — and the honest answer lives in research papers nobody reads. Marketing says yes to everything; the evidence is nuanced, sometimes contradictory, and buried in 93k abstracts. A question like "does magnesium help you sleep?" deserves the studies, cited, not a confident paragraph.

## Corpus — where the data comes from

- **PubMed dietary-supplements slice** — `("dietary supplements"[mesh] OR "dietary supplement*"[tiab]) AND 2015:2026[pdat]` = **93,572 articles** (verified live). Title, abstract, MeSH terms, publication type, journal, date. Public domain, free API, no key required.
- **Evidence-grade filter** — randomized controlled trials, meta-analyses, systematic reviews, and controlled trials shrink the slice to **17,698 abstracts → 17,696 indexed chunks** (measured). Every indexed document is answer-grade evidence.
- **The famous benchmark didn't fit — measured, not assumed.** BioASQ's golden set looked like the obvious eval; checking its 40,181 evidence passages against the slice found 41 (0.1%), and essentially zero of its 4,719 questions are about supplements. So the golden set is authored and verified instead: each question enters only after its evidence PMIDs are identified in the corpus — domain-curated, with the receipts committed per row.
- **Withheld abstracts = real abstention cases** — some evidence-grade abstracts stay out of the index; questions that need them must abstain. Refusal is tested, not hoped for.
- **Stays fresh:** the downloader keeps a ledger of fetched PMIDs, so re-running it pulls only new articles — PubMed won't page past 10k records, so acquisition slices the query into monthly windows and unions them.

## The finding that shaped the design

**Coverage is not evidence.** A semantic judge pass over the full 4,026-question admitted pool confirmed real supporting evidence for only 24% — **52% of everything lexical coverage admitted has no supporting abstract at all.** A system that answers whatever it retrieves fabricates on half its input. So this one verifies every claim before text is trusted: deterministic citation + number checks, per-claim groundedness (≥0.7), verdict scored against the combined evidence — a failed check marks its sentence `unverified` on the answer rather than being silently rewritten, only unparseable output abstains, and an unclear verdict gets re-scored against the pack (refusals are checked too, not just answers). The judge being down fails closed, never open.

## Shape

Question → safety gate (injection + intent) → keyword + vector search → RRF fusion + rare-entity rescue → verbatim sentence extraction (entity-screened on rare substances) → deterministic + per-claim judge verification → cited answer, abstain, decline, or refuse.

- **Advice gate — the stage a health corpus demands:** "does vitamin C help colds?" is an evidence question the system answers. "Should I take it? How much? With my medication?" is medical advice the system declines by design. Prompt-injection attempts are refused at the same gate.
- **Bounded corpus, visible boundary:** dietary-supplement research, 2015–2026 — plus a thin incidental complementary-medicine slice (~0.1% of chunks) that abstains honestly when evidence is thin.
- **Two answer paths, same checks:** `answer()` buffered — unverified text never leaves (the eval path). `answer_events()` streamed — extraction progress plus per-claim judge stamps emitted live. An opt-in writer lane (`EXTRACTIVE=0`) synthesizes prose instead of quoting and streams its tokens the same way. The event contract is what the local exhibit serves over SSE — and what the deployed Worker streams at luisosio.dev/sr/v1/, unchanged.
- **Exact citations:** answers link PMID, journal, year, and publication type — retrieved, never generated.

## Measured on the golden set — and a held-out set it never saw

- **91%** of verified-evidence rows answered with citations on the golden set (360 runs at the time — **0 status flips**, every row gives the same outcome).
- **0** dangerous answers and **100%** refusal on refuted rows — the verification layer holds where it matters.
- **92%** verified-answer rate on a **199-row held-out set** — disjoint questions drawn from the same admitted pool, used only to measure, never to tune.
- **60 tests** green; every stage rerunnable and inspectable alone (`python -m` per module).

## Honest classification

A **deterministic pipeline, not an agent** — the code path is fixed: gate, retrieve, fuse + rescue, write, verify, answer-or-abstain. The model writes and scores at fixed points; it never chooses the next step. Same architecture family as the private production work — the corpus changed, the discipline didn't.

## What could go wrong

- **A hallucinated health claim is worse than a wrong tax article** — the advice gate + verify-before-trusted contract are the safety surface, not polish.
- **Invented PMIDs or doses** → citations are retrieved abstracts only; numbers must exist in a cited source (spelled-out numbers and rounding handled).
- **Right topic, wrong substance** → retrieval finds lexically-near compounds (asked *myrcia*, found *myricetin*), so a dedicated entity check scores each candidate source — exact identity or established name only, constituents and derivatives don't count — before a quote can enter the answer.
- **Out-of-corpus questions** → bounded slice + withheld abstracts make "not covered" decidable and tested.
- **MeSH indexing lag** → the newest articles aren't MeSH-tagged for weeks; the query ORs title/abstract terms so recent research isn't silently dropped.
- **Abstracts, not full text** — a stated limitation; PMC open-access full text is the documented v2 path.

*Fully open source — code, corpus scripts, eval harness, and results land together. This page cites measured numbers, not invented metrics.*
