Generated from folio-assistant-core/processes/library/ingest-extract-structure.bpmn by gen-processes-viz.ts — do not edit here. All processes

Ingestion subprocess — extract structure

Process_ExtractStructure · advisory · 5 step(s)

Extract the structure of an uploaded source: its text layer, OCR where it has none, sections, structure and claim candidates. folio-assistant — Ingestion subprocess — extract structure.

Source of truth: this file. Open it in bpmn.io, Camunda Modeler, or any other BPMN 2.0 tool. The SVG under docs/assets/img/workflows/ is generated from it by bun run cat render:bpmn — never hand-edit the SVG.

The extension on an activity names the folio-assistant skill that implements it; marks a step that reads or writes the shared work plan in beans/.

BPMN diagram: Ingestion subprocess — extract structure

How it connects

Called by: L1 document ingestion — a document to the L1 source knowledge graph
Calls: none
Presented on: no docs page section shows this diagram

Lanes — who acts

lane	role	what it does here
Ingestion Engine (agent, runs unattended)	`ingestion-agent`	Two provenance paths converge on the same Task_Sections output — an embedded text layer extracted directly, or OCR that leaves the fuller text in ocr/ with only a stub in sections/ — and which branch this lane took is invisible downstream except to a grep that knows to check the fourth tier. Task_Candidates is the last step and it stays a proposal: theorems and definitions extracted here are never adjudicated verdicts, so nothing downstream may treat this lane’s output as settled.

Steps

Every one of the 5 step(s) is documented.

step	lane	skill / sub-process	what it does
Extract the text layer `Task_ExtractText`	Ingestion Engine (agent, runs unattended)	`document-intake`	The document has a text layer: extract it with pdf-extract (pdfminer.six, then a zero-dependency content-stream reader). Exit 2 means no text layer — a scan — and routes to OCR; it is not an empty document.
*OCR to ocr/page-.txt** `Task_Ocr`	Ingestion Engine (agent, runs unattended)	`document-intake`	A document ingested this way keeps its text in ocr/ and only a stub in sections/, which the documented sections/ grep cannot see. That asymmetry is why the corpus-grep checklist has a fourth tier. The run also records WHICH SOFTWARE made the pages, in ocr/run-context.json: a folio-run-context/v1 (cat-harness schemas/run-context.ts, the record test runs and QA reports use for the same question), naming the OCR engine, the rasteriser and each language model with its version and the sha256 of the bytes that ran. Two OCRs of one scan can read differently (“Manual” and “Manval”, bean turh), and without it neither could be told from the other or reproduced.
*Split into sections/.md with doc_brief front-matter** `Task_Sections`	Ingestion Engine (agent, runs unattended)	`document-intake`	Contextual retrieval: a chunk in isolation loses what makes it mean anything, so each section carries the document brief.
Write structure.json (TOC, page ranges, metadata) `Task_Structure`	Ingestion Engine (agent, runs unattended)	`document-intake`	Write structure.json (pdf-structure/v1): doc id, TOC from the PDF outline or inferred from headings, page ranges, metadata, and source{} with the sha256 and a mimetype sniffed from the bytes, never the extension. A structure that could not be determined says so in structure_note rather than being rendered as one. For OCR’d text, source.runContext cites the OCR step’s run context by sha256. A title already known for the document (the catalogue’s dc.title, or one given with –title) is passed through as the raw title, so the sections’ doc_title does not carry a scan’s misreading (bean turh).
Extract claim candidates `Task_Candidates`	Ingestion Engine (agent, runs unattended)	`document-intake`	candidates.json holds extracted theorems and definitions. Proposals only – never adjudicated verdicts.

Decisions

Every one of the 1 decision(s) is documented.

decision	what decides it	branches
Embedded text layer? `Gateway_HasText`	Does the binary carry an embedded text layer? `yes` extracts it; `no — scanned` runs OCR into ocr/page-*.txt.	yes → Extract the text layer no — scanned → OCR to ocr/page-*.txt