Evidence for a recommendation
On this page
This page is generated from docs/source/process-evidence/ — each section below links to its own source.
How a recommendation in a guideline gets its evidence — who looks, where they look, and what makes a source authoritative rather than merely cited.
STRAWPERSON. The process shape below is real, renders, and passes the workflow policy gate. What is deliberately not settled here: which registries count as trusted, which grading system a folio declares, and how the observatory APIs are bound. A separate process refines requirements; this exists so there is something concrete to refine.
Why this is a subprocess of editing
It runs inside the editing loop, between claiming the bean and drafting the change — see Editing and the HCI validation gate. That position is the whole point: the evidence is gathered before the sentence is written, not assembled afterwards to support one.
A recommendation drafted first and evidenced second is a literature search for confirmation. The step order is what makes it not that.
It is a call activity, so it has its own file, its own lanes and its own diagram, and the editing diagram stays readable. It also has nothing to do for content that carries no recommendations — a mathematics folio has none — and is skipped there rather than satisfied vacuously.
The subprocess
Four lanes, because four different kinds of actor are involved and conflating them is how accountability gets lost:
| lane | who | owns |
|---|---|---|
| Author / guideline developer | person | the PICO, the appraisal, the grade |
| Evidence agent | system | retrieval across the three classes |
| Trusted registries and observatories | external | whether a citation resolves |
| Work plan — beans | shared | every gap and every unverified citation |
The third lane is external on purpose. Authority is not something the folio can assert about itself; it is a fact resolved at the publisher. Drawing that lane inside the system boundary would make “authoritative” a self-assessment.
BPMN 2.0 source · full-size SVG
Step one looks inward
The first step looks inward, not outward.
Before any search, the author reads what the document, DAK or IG already says on this question. A new recommendation that contradicts a sibling recommendation in the same artefact is a defect that no amount of external evidence will reveal — the sources can all be impeccable and the guideline still incoherent.
This is also the cheapest step by far, and it is the one most often skipped.
Framing the question as PICO
Population, Intervention, Comparator, Outcome.
WHO guideline development states the question this way before any retrieval, and the reason is auditability rather than ceremony: without a PICO, “we looked for evidence” is not a claim anybody can check. With one, the search is reproducible — a second reader can run it and say whether the same body of evidence comes back.
The PICO is carried on the recommendation, not discarded after the search. It is what a later reader needs in order to tell whether new evidence bears on this recommendation or a different one.
Three classes of evidence, not one
Retrieval fans out across three classes that are not interchangeable, and keeping them apart is the point of the parallel gateway:
Trusted sources — L1. Ingested primary literature under library/: the
sections/*.md tree and, where a document was scanned, ocr/page-NNN.txt.
Known blind spot. A document that was OCR’d but never re-run through
pdf-structure.pyhas only a stub insections/, so a sections-only search misses it entirely. Measured: 26library/*/ocr/trees against 11structure.jsonrecording"text_source": "ocr". See Document ingestion.
Trusted content — L2 DAKs, L3 IGs. Already-adjudicated guidance: WHO SMART Digital Adaptation Kits at L2, FHIR Implementation Guides at L3. A recommendation adopted in a published DAK is evidence of a different kind from a primary study — it is a prior decision, with its own provenance and its own grading — and must be cited as that. Laundering it into primary evidence is the failure mode this separation exists to prevent.
Data repositories and statistical datasets. Indicator series and microdata. Distinct again because the unit of evidence is a measurement with a population, a period and a method attached — not a claim in prose.
Verifying authority, not asserting it
Each candidate is resolved against the standard API of the body that publishes it, so that “authoritative” is a resolved fact rather than an assertion in a citation. Four things are checked:
- the identifier resolves at the publisher;
- the retrieved record matches what the citation claims;
- the version and date are the ones cited;
- the publisher is one this folio has declared trusted.
Check 2 is the one that catches real errors. A citation can resolve perfectly and still not say what the sentence citing it says.
An unresolvable citation is neither dropped nor silently kept. It becomes a bean, so the gap is visible in the work plan instead of surviving as a footnote nobody re-checks.
Not settled here, and deliberately so: the registry list, the endpoints, and the shape of the stored authority record. This container has no outbound network, so no endpoint below the level of “the publisher’s own API” has been verified, and naming specific ones would be writing down a guess. That is a requirements question, and there is a separate process for it.
Grading the body, and recording a gap
The grade attaches to the body of evidence for one PICO question — which is why appraisal sits after the join and not inside the fan-out. A single citation does not carry a certainty rating; a body of evidence answering a question does.
Then one gateway: is this sufficient for a recommendation?
Insufficient evidence is a result. It is recorded as an honest gap the guideline can state — “the evidence does not support a recommendation here” — rather than being closed by weakening the recommendation until the available evidence happens to support it. That move is always available and it is always wrong, which is why the “no” branch has somewhere to go.
On the “yes” branch the PICO, the graded body of evidence, and every verified citation with its resolved authority record are written onto the recommendation block. A reader can then retrace the chain from the sentence back to the source.
What is not built
Everything below is absent, not partially working. The diagram is a model of the intended process; these are the parts with no code behind them.
| state | |
|---|---|
| the trusted-registry list | not decided |
| observatory API bindings | not written; no endpoint verified from this container |
| the authority record’s schema | not designed |
| the grading system a folio declares | not chosen |
an evidence-retrieval skill | does not exist — the diagram names it, nothing implements it |
| PICO fields on a recommendation block | not in the schema |
The diagram naming a skill that does not exist is deliberate and is the same convention the ingestion diagrams use: the model says what would implement a step, and the gap between the model and the code is visible rather than hidden.