Folio Assistant Core

Process

Ingestion subprocess — extract structure

https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure

JSON-LD · diagram · in graph folio-assistant-core.jsonld ·

nameIngestion subprocess — extract structure
summaryExtract the structure of an uploaded source: its text layer, OCR where it has none or where a page's text layer fails the page check, sections, structure and claim candidates.
descriptionExtract the structure of an uploaded source: its text layer, OCR where it has none or where a page's text layer fails the page check, sections, structure and claim candidates. Every step is an arm or rung of bun run cat ingest (cat-harness-tools scripts/ingest-document.ts).
enforcementadvisory
sourcePathprocesses/library/ingest-extract-structure.bpmn
sourceUrlhttps://github.com/litlfred/folio-assistant-core/blob/main/processes/library/ingest-extract-structure.bpmn
depictionhttps://litlfred.github.io/folio-assistant/assets/img/workflows/ingest-extract-structure.svg
startNodeBinary in hand
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/StartEvent_Binary
nodeCount9
flowCount9
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

Flows

flow/Flow_e1

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromBinary in hand
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/StartEvent_Binary
toEmbedded text layer?
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Gateway_HasText
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e2

nameyes
partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromEmbedded text layer?
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Gateway_HasText
toExtract the text layer
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_ExtractText
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e3

nameno — scanned
partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromEmbedded text layer?
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Gateway_HasText
toOCR to ocr/page-*.txt
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Ocr
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e4

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromExtract the text layer
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_ExtractText
toCheck each page; OCR pages that fail
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_PageCheck
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e9

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromCheck each page; OCR pages that fail
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_PageCheck
toSplit into sections/*.md with doc_brief front-matter
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Sections
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e5

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromOCR to ocr/page-*.txt
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Ocr
toSplit into sections/*.md with doc_brief front-matter
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Sections
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e6

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromSplit into sections/*.md with doc_brief front-matter
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Sections
toWrite structure.json
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Structure
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e7

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromWrite structure.json
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Structure
toExtract claim candidates
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Candidates
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes

flow/Flow_e8

partOfIngestion subprocess — extract structure
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure
fromExtract claim candidates
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/Task_Candidates
toStructured
https://litlfred.github.io/folio-assistant/folio-assistant-core/process/Process_ExtractStructure/node/EndEvent_Structured
inSubgraphcore-processes
https://litlfred.github.io/folio-assistant/folio-assistant-core/directory/core-processes