Generated from
who-iris/processes/iris-cdn-initial-load.bpmnbygen-processes-viz.ts— do not edit here. All processes · who-iris processes
IRIS CDN initial load
Process_IrisCdnInitialLoad · strict (defaulted) · 10 step(s)
The first full export of WHO IRIS onto Cloudflare (git on Artifacts, bytes in R2): every item record and every bitstream: metadata plus PDFs. Run in batches with a checkpoint, so a run that stops resumes where it stopped and never re-fetches what it already holds. Ends in one signed, verified release.
How it connects
- Called by: no call activity names this process
- Calls: Sign and publish an IRIS CDN release, Package and distribute partitioned W3C N-Quads and named queries
- Presented on: no docs page section shows this diagram
- Running instances: none recorded
Lanes — who acts
Steps
3 of 10 step(s) carry no documentation — activity-documented lists them.
| step | lane | skill / sub-process | what it does |
|---|---|---|---|
State scope and budget, with denominatorsTask_StateBudget | Owner | iris-cdn-exporter | Items, files and bytes upstream (273,559 items, 1,057,223 files, 361.55 GiB as last transcribed), the storage target for each kind of file, and a request rate IRIS will tolerate. A budget with no denominator is a preference, not a limit. |
Plan batches by community and collectionTask_PlanBatches | IRIS CDN Exporter (agent) | iris-cdn-exporteriris-dspace | One batch per collection (large collections split by page). |
Harvest a batch of item records (REST)Task_HarvestBatch | Build pipeline | iris-rest-feediris-dspace | DSpace REST: the item, its metadata, bundles and bitstream descriptors. Written as canonical JSON (sorted keys, LF, no fetch timestamp inside the record) so the same item always serialises to the same bytes. |
Fetch bitstreams not already heldTask_FetchBitstreams | Build pipeline | iris-cdn-exporter | A bitstream is stored under its SHA-256. If that object already exists it is not downloaded again; a re-run of a batch therefore moves no bytes. |
Store bytes under SHA-256 (skip if held)Task_StoreBytes | Build pipeline | iris-cdn-exporter | — |
Commit the batch checkpointTask_Checkpoint | Build pipeline | iris-cdn-exporter | state/initial-load.json in the Cloudflare git repository. A restarted load reads it and skips finished batches. |
Quarantine and report the mismatchTask_Quarantine | Build pipeline | iris-cdn-exporter | Not published, not retried silently. The record keeps its reference to IRIS and the bitstream is marked unknown. |
Build the graph partitionsCall_BuildGraph | Build pipeline | calls Package and distribute partitioned W3C N-Quads and named queriesiris-oxigraphnquads-distribution | The platform process nquads-distribution (cat-harness): spine plus one partition per community, zero blank nodes, sorted N-Quads. |
Sign and publish the releaseCall_SignPublish | Build pipeline | calls Sign and publish an IRIS CDN releaseiris-cdn-signing | — |
Review the load report; open beans for gapsTask_ReviewLoad | IRIS CDN Exporter (agent) | iris-cdn-exporter | — |
Decisions
1 of 2 decision(s) carry no documentation — gateway-documented lists them.
| decision | what decides it | branches |
|---|---|---|
Bytes match the IRIS checksum?GW_Fixity | DSpace reports an MD5 checksum per bitstream. The downloaded bytes must match it before their SHA-256 is recorded. | yes → Store bytes under SHA-256 (skip if held) no → Quarantine and report the mismatch |
More batches?GW_More | — | yes → Harvest a batch of item records (REST) no → Build the graph partitions |