CRDM methodology
On this page
- What is CRDM?
- Roles — who does what
- Why CRDM for folio-assistant
- When the CRDM workflow activates
- The process
- Phase 1 — Needs assessment
- Phase 2 — Business process analysis
- Phase 3 — Requirements definition
- Phase 4 — Impact analysis and migration planning
- Phase 5 — Sign-off and bean creation
- Phase 6 — Iterative MVP development and stakeholder review
- Agent skills and tooling
- What is not built yet
This page is generated from docs/source/process-crdm-methodology/ — each section below links to its own source.
How folio-assistant uses the Collaborative Requirements Development Methodology (CRDM) — a structured, participatory framework from the Public Health Informatics Institute — to manage feature requests, platform changes, and content-pipeline evolution. When a user, reviewer, or editor asks for something that is not content but capability — a new block kind, a rendering change, a QA criterion — the agent enters a CRDM-guided workflow rather than improvising a solution.
What is CRDM?
The Collaborative Requirements Development Methodology (CRDM)™ is a structured, facilitated approach developed by the Public Health Informatics Institute (PHII) to help organisations document workflows and define functional requirements for information systems. Rather than starting with technology, the methodology prioritises understanding the underlying business processes first.
CRDM organises work into three primary areas of concentration:
Business Process Analysis (BPA) — document and analyse how the organisation currently performs its work. Understand manual workflows, identify bottlenecks, clarify how data is collected and used.
Business Process Redesign (BPR) — once the current state is understood, identify areas for improvement. Streamline operations and design a more efficient future-state workflow that the new capability will support.
Requirements Definition — define specific functional requirements for the information system based on the redesigned business processes. Ensure the result is fit-for-purpose, scalable, and interoperable.
The key insight is “begin with the end in mind”: technical requirements must be intentionally aligned with achieving specific outcomes rather than reacting to a feature request at the surface level.
References:
- PHII — CRDM — the canonical description
- PATH — Collaborative Requirements Development Methodology — the PATH adaptation for global health informatics
- Issue #203 — the request that initiated this integration
Roles — who does what
Three distinct roles participate in a CRDM cycle. The boundaries between them matter — collapsing the first two into one person loses the quality gate that makes the process work.
Feature Requestor / Business Analyst (BA)
The person who identifies a need and initiates the CRDM cycle. In folio-assistant this is typically a content author, editor, or programme lead who encounters a gap in the platform while doing their work.
The BA is also the coordinator of the CRDM process:
- Initiates — describes the need in a GitHub issue or chat session.
- Interacts with the agent — the BA is the agent’s primary counterpart. They refine needs, review synthesised requirements, negotiate scope, and decide what constitutes a viable MVP.
- Coordinates with stakeholders — the BA understands the broader context: who else is affected, what workflows will change, whose sign-off is needed. They bring stakeholders in at the right moments (review, testing, approval) rather than expecting them to participate in every iteration.
- Accepts or rejects iterations — during Phase 6, the BA reviews each increment the agent delivers, decides whether it meets the requirement, and either accepts it or requests changes.
The BA does not need technical expertise. They need domain expertise — they know what the feature should do, not how it should be built. The agent handles the technical design and implementation.
Stakeholders
The people affected by the feature who must review and approve it, but who are not in the day-to-day development loop with the agent. Examples:
- A review committee that must approve changes to a guideline’s normative content.
- Country programme managers who will use the localisation feature in their context.
- IT teams who must deploy and maintain the result.
- End users who will interact with the new capability.
Stakeholders participate at defined checkpoints, not continuously:
| Checkpoint | What stakeholders do |
|---|---|
| Needs review (Phase 1) | Confirm the BA’s needs statement reflects their reality |
| Requirements sign-off (Phase 5) | Approve that the requirements are correct and complete |
| MVP testing (Phase 6) | Test the delivered feature in their context, report findings |
| Feature sign-off (Phase 6) | Confirm the feature meets their needs; close the issue |
The BA bridges the gap: stakeholders should never need to read a PR, understand a bean, or interact with the agent directly.
Agent
The AI assistant that implements the CRDM process and builds the feature. The agent:
- Facilitates — guides the BA through the CRDM phases, synthesises inputs, proposes requirements, identifies impacts.
- Implements — writes code, creates tests, opens PRs, posts summaries.
- Iterates — responds to BA feedback, adjusts implementation, re-runs validation.
- Documents — maintains the GitHub issue as the single source of truth for stakeholders, posts implementation summaries, updates documentation.
The agent interacts with the BA only, not with stakeholders directly. If stakeholders have feedback, it flows through the BA who translates it into actionable direction for the agent.
Stakeholders ──review/approve──▶ BA ──directs──▶ Agent
◀──delivers────┘
Why CRDM for folio-assistant
folio-assistant is a platform, not an application with a fixed feature set. Authors, editors, and reviewers working in folio repositories regularly discover needs that are not content — they are capability gaps in the platform itself: a missing block kind, a QA criterion that does not exist, a rendering mode that does not handle their document structure, a workflow step that has no tooling.
Without a methodology, agents improvise. The history of this repository shows what that looks like: a feature request arrives as a chat message, the agent builds something that addresses the immediate symptom, and the result does not compose with the rest of the platform — because nobody mapped the current workflow, identified who else is affected, or wrote the requirements down before coding started.
CRDM solves this by inserting a structured requirements phase between “I need this” and “here is the code”. The methodology is a natural fit because folio-assistant already has the primitives it needs:
| CRDM concept | folio-assistant primitive |
|---|---|
| Stakeholder identification | <name>.config.json roles, GitHub CODEOWNERS |
| Business process documentation | BPMN workflow diagrams under processes/ |
| Requirements artefact | GitHub issue with structured fields |
| Work-plan items | beans — the single todo mechanism |
| Impact analysis | Content graph (content-graph.ts), schema constraints, QA registry |
| Iterative review | PR-based feedback, content_validate, qa_sweep |
The goal is not to add ceremony. It is to make the agent recognise when a request is a feature rather than content, and shift into a requirements-gathering mode rather than jumping straight to implementation.
When the CRDM workflow activates
An agent enters the CRDM workflow when it recognises that a request is about platform capability rather than folio content. The triggers are:
Explicit feature request — the user says “I need a new block kind”, “add a QA check for X”, “the renderer should support Y”. These are unambiguous.
GitHub issue tagged as a feature or enhancement — when the agent is asked to resolve an issue whose labels or body describe new functionality rather than a content edit.
Discussion-derived need — during an authoring session, the user encounters a limitation. The conversation shifts from “write this chapter” to “the platform should be able to do X”. The agent detects this shift.
Schema or pipeline change — a request that would modify files under
schemas/,content/pipeline/,adapters/, orsrc/rather than under a folio’scontent/<paper>/.Cross-cutting concern — the request affects multiple folios or multiple content types. A single-folio content edit does not trigger CRDM; a change to how all folios validate does.
The detection rule: if implementing the request would require changes to folio-assistant (the platform repository) rather than to a folio repository, the request is a feature, and the agent should enter CRDM rather than coding directly.
When the agent detects a CRDM trigger, it should:
- Acknowledge — tell the user it has recognised a feature request
- Cite the source — link to the issue, message, or conversation turn
- Propose entering the CRDM workflow — explain what that means (structured requirements gathering before implementation)
- Get consent — the user may prefer to skip the methodology for small changes; the agent respects that but notes the risk
The process
The diagram above shows the full CRDM workflow as a BPMN 2.0 collaboration with three swim lanes:
BA / Feature Requestor — initiates the request, reviews needs and requirements, signs off, reviews each increment the agent delivers, decides when to share an MVP with stakeholders, translates stakeholder feedback into agent-actionable direction, and confirms final delivery.
Agent — detects the feature request, scans for matching issues, runs through the six CRDM phases, creates beans, implements on feature branches, posts summaries to the issue.
Stakeholders — confirm needs, approve requirements, test the MVP in their own context, and do final feature sign-off on the issue.
Three feedback loops are visible:
Needs loop (Phase 1) — the agent synthesises needs, the BA reviews, stakeholders confirm. If revisions are needed, the agent re-synthesises.
Requirements loop (Phases 3–4) — the agent defines requirements and impact analysis; the BA reviews, stakeholders approve; iterate until approved.
Dual development loop (Phase 6):
- Inner loop (BA ↔ Agent) — the agent implements an increment, the BA tests it against acceptance criteria. Fast — multiple iterations per session.
- Outer loop (BA → Stakeholders) — when the BA judges enough increments constitute a testable MVP, stakeholders test it. Findings flow back through the BA to the agent. Slower — stakeholder cadence.
The BA is the bridge: they keep the agent productive in the inner loop while waiting for stakeholder availability in the outer loop.
See the agentic harness page for how this workflow fits into the broader agent–user interaction model.
Phase 1 — Needs assessment
The agent helps the user articulate what they need and why, before anyone discusses how to build it. This is where most ad-hoc requests go wrong — the solution is specified before the problem is understood.
What the agent does in this phase:
Identify the requester and stakeholders — who is asking, who else is affected? The word-document-import request (#197) came from one person but affects anyone doing public consultation review, HRH handbook adaptation, or PCMT traceability. The agent should surface those connections.
Synthesise the need from multiple sources — a requirement rarely arrives as a clean statement. It comes from a chat message, a GitHub issue, a discussion thread, a review comment. The agent gathers these fragments and presents a consolidated statement of need back to the user for validation.
Suggest other stakeholders — if the request touches a workflow that others use (e.g. country adaptation, formal review processes), the agent should identify who else should weigh in and propose inviting them.
Document the “why” — every requirement gets a rationale. Not “import Word documents” but “enable public consultation comment triage through a structured ingestion pipeline, so that SAG members can review and approve changes through a formal process.”
Deliverable: a GitHub issue (or issue comment) with:
- Statement of need
- Identified stakeholders
- Rationale linking the need to a concrete workflow
- References to the conversation or discussion that surfaced the need
Phase 2 — Business process analysis
Before changing the platform, understand how the work is done today. The agent maps the current workflow — manually if needed — and identifies where the gap actually sits.
What the agent does in this phase:
Map the current workflow — use the existing BPMN diagrams under
processes/as a starting point. If the affected workflow is already diagrammed (e.g. the content lifecycle, the publication pipeline, the document ingestion flow), read it and identify the specific activity or decision point where the gap appears.Identify bottlenecks and pain points — the request “import Word documents” might mask a deeper problem: “the public consultation feedback arrives as an Excel spreadsheet and a Word document, and there is no way to triage 200 comments without manually reading each one.” The agent should probe for the real bottleneck.
Check existing capabilities — before proposing new tooling, verify what already exists. folio-assistant’s ingestion pipeline already handles PDFs, images, and structured text. Does the request require a new capability or an extension of an existing one?
Document the current state — write up the as-is workflow, either as prose in the issue or as a BPMN fragment. This becomes the baseline against which the redesign is measured.
Deliverable: a current-state workflow description attached to the issue, identifying exactly where the capability gap sits.
Phase 3 — Requirements definition
Translate the understood need into specific, actionable requirements with
proposed tooling descriptions. All requirements should be developed fully, with
proposed .ts skills, software tooling descriptions, and requirements for
folio-assistant SOPs.
What the agent produces in this phase:
- Functional requirements — each stated as a testable capability:
- “The ingestion engine SHALL detect
.docxfiles inuploads/and extract semantic structure (headings, lists, tables) into Markdown.” - “The review triage tool SHALL present comments grouped by document section and allow batch accept/reject/defer decisions.”
- “The ingestion engine SHALL detect
- Proposed skills and tooling — for each requirement, a concrete
description of the skill or tool that implements it:
- Skill name, description, and which adapter it belongs to
- Input/output contract
- MCP tool registration (if applicable)
- QA criteria (if applicable)
Schema changes — if the requirement needs new types, block kinds, constraint rules, or configuration fields, these are specified here with their Zod schemas.
Pipeline changes — if the requirement needs new pipeline scripts under
content/pipeline/, new validators, or new renderers, the file names and responsibilities are listed.- Acceptance criteria — how we know the requirement is met. Each criterion should be mechanically testable where possible.
Deliverable: a requirements document in the GitHub issue, structured as a checklist of capabilities with their proposed implementations. Each requirement links back to the need (Phase 1) and the workflow gap (Phase 2) it addresses.
Phase 4 — Impact analysis and migration planning
Before implementation begins, assess what the proposed changes affect across the platform and any active folios.
What the agent analyses:
Content impact — does the change affect existing folio content? If a schema changes, which folios need migration? The content graph (
content-graph.ts) and the constraint rules (schemas/constraints.ts) make this measurable.Pipeline impact — which pipeline scripts are affected? Does the change require new validators, new renderers, or modifications to existing ones? List every file under
content/pipeline/that would be touched.QA impact — does the change add, modify, or remove QA criteria? How does it affect the existing QA sweep results? Any criterion change must be registered in
qa-criteria-registry.ts.Adapter impact — does the change affect one adapter (paper, document, dak) or all? Changes to
DocumentContentAdapterripple intoPaperContentAdapterbecause it extends it.- Migration plan — if existing content or configuration must change, write
the migration steps. Include:
- Which folios are affected
- What the codemod or manual migration looks like
- What breaks if the migration is not run
- Rollback strategy
- Test plan — what tests need to be added or updated? Which existing tests might break?
Deliverable: an impact assessment section in the GitHub issue listing every affected file, folio, and workflow, with a migration plan if applicable.
Phase 5 — Sign-off and bean creation
The requirements and impact analysis are presented to the user (and any identified stakeholders) for review and sign-off. This is the gate between requirements and implementation.
The sign-off process:
- Present the consolidated requirements — the agent produces a summary
comment on the GitHub issue containing:
- The need statement (Phase 1)
- The current-state workflow and gap (Phase 2)
- The requirements checklist (Phase 3)
- The impact assessment and migration plan (Phase 4)
Solicit feedback — ask the user and stakeholders to review. The review happens on the issue, not in chat — so it is visible, linkable, and survives beyond any single session. This is the transparent review process: comments, questions, and revisions are all recorded in the issue thread.
Iterate — if the review surfaces new requirements, missing stakeholders, or concerns about the impact, cycle back to the relevant phase. The issue thread records each iteration.
- Create beans — once the requirements are agreed, the agent creates
beansfor each unit of implementation work. Each bean:- References the parent issue
- Has a clear title describing the deliverable
- Is scoped to a single PR-sized unit of work
- Is created using the check-before-create protocol to avoid duplicates
- Record the decision — the sign-off comment on the issue serves as the decision record. It states what was agreed, what was deferred, and links to the beans that will implement it.
Deliverable: a sign-off comment on the issue, plus beans for each work item.
Phase 6 — Iterative MVP development and stakeholder review
Once requirements are signed off and beans are created, the agent and BA enter the iterative MVP development cycle. This is where the feature gets built, tested, and refined until stakeholders accept it.
The inner loop: BA ↔ Agent
The BA and agent work together in tight iterations:
┌──────────────────────────────────────────┐
│ INNER LOOP │
│ │
│ Agent BA │
│ ┌─────────┐ PR + demo ┌──────┐ │
│ │Implement├───────────────►│Review │ │
│ │ (bean) │ │ + test│ │
│ │ │◄───────────────┤ │ │
│ └─────────┘ feedback └──────┘ │
│ │
└──────────────────────────────────────────┘
- Agent claims a bean — implements the requirement on a feature branch, opens a PR, posts a summary to the issue.
- BA reviews — tests the feature, compares it against the requirement and acceptance criteria from Phase 3. The BA does not need to read code; they test the behaviour.
- Feedback or accept — the BA either requests changes (the agent iterates) or accepts the increment.
- Repeat until the bean is resolved.
Each bean produces a minimum viable increment — not the whole feature, but one requirement satisfied. The BA decides when enough increments have accumulated to show stakeholders.
The outer loop: BA → Stakeholders
When the BA judges that the delivered increments constitute an MVP worth testing, they bring stakeholders in:
┌──────────────────────────────────────────────────────────────┐
│ OUTER LOOP │
│ │
│ BA Stakeholders │
│ ┌──────────┐ MVP demo ┌────────────────┐ │
│ │Accumulate├─────────────►│Test in context │ │
│ │increments│ │(their workflow)│ │
│ │ │◄─────────────┤ │ │
│ └──────────┘ findings └────────────────┘ │
│ │ │
│ ▼ │
│ Translate findings into │
│ agent-actionable feedback │
│ (new beans or bean updates) │
│ │ │
│ ▼ │
│ Re-enter inner loop │
└──────────────────────────────────────────────────────────────┘
- BA shares the MVP — the staging URL, a demo, or a walkthrough. The BA frames it: “here is what we built, here is what it does, here is what we need you to test.”
- Stakeholders test — in their own context, with their own data, against their own workflows. They report what works and what does not.
- BA translates findings — stakeholder feedback is domain language
(“the table doesn’t show the right columns”). The BA translates it
into actionable direction for the agent (“the
review-triageview needs columns X, Y, Z from the feedback spreadsheet”). - Agent iterates — new beans or updates to existing beans. The inner loop resumes.
- Repeat until stakeholders are satisfied.
The exit: feature sign-off
When all beans for the feature are resolved and stakeholders confirm the MVP meets their needs:
- BA confirms — all acceptance criteria from Phase 3 are met.
- Stakeholders sign off — on the GitHub issue, not the PR.
- Agent closes the issue — with a delivery summary listing what was built, what was deferred, and any new issues opened for future work.
- Documentation updated — the agent updates the docs pages and BPMN diagrams to reflect the new capability.
Why two loops, not one
The inner loop is fast — the BA and agent can iterate multiple times per session. The outer loop is slow — stakeholders have other work, need time to test, and may only be available weekly.
Collapsing them into one loop would either:
- Slow the agent down to the stakeholder cadence (wasting agent time), or
- Rush stakeholders through testing (producing shallow feedback).
The BA absorbs the impedance mismatch. They keep the agent productive in the inner loop while waiting for stakeholder availability in the outer loop. Between outer-loop checkpoints, the agent works on other beans or other features.
Agent skills and tooling
The CRDM workflow is supported by skills and tools that the agent uses at each phase. Some exist today; others are proposed.
Existing capabilities
| Capability | Used in phase | How |
|---|---|---|
| GitHub issue management | All phases | Issues are the primary artefact for requirements, impact analysis, and sign-off. Agents reach GitHub through the MCP tools, not the gh CLI, which is not available in every harness |
BPMN workflow authoring (skill: bpmn-authoring) | Phase 2 | Agent can create and modify BPMN 2.0 process diagrams |
Content graph (content-graph.ts) | Phase 4 | Dependency analysis across blocks and chapters |
QA criteria registry (qa-criteria-registry.ts) | Phase 3, 4 | Register and scope new QA checks |
Beans (beans CLI) | Phase 5, 6 | Work-plan creation and tracking |
Content validation (content_validate MCP tool) | Phase 6 | Verify changes against schema and constraints |
Schema types (schemas/*.ts) | Phase 3 | Define new types, constraints, and builders |
Document ingestion (uploads/ → library/ pipeline) | Phase 2 | Ingest source documents for review |
Skills that now exist
| Skill | Phase | Purpose |
|---|---|---|
crdm-detect | Trigger | Recognise when a request is a feature rather than content. Five categories of detection phrasing, an explicit “not a feature request” list, and the session-state rules |
crdm-requirements-workflow | All phases | The six-phase process, the actors table, and the issue-association rules |
Both are guidance, not instrumentation: they tell an agent what to do, and nothing checks that it did. See “What is not built yet” for why that distinction is kept explicit.
Proposed skills
| Skill | Phase | Purpose |
|---|---|---|
crdm-needs-assessment | Phase 1 | Guide the user through needs articulation; identify stakeholders |
crdm-impact-analysis | Phase 4 | Automated scan of affected schemas, pipeline scripts, adapters, and folios |
crdm-requirements-template | Phase 3 | Generate structured requirements from conversation |
review-triage | Phase 6 | Triage review comments from ingested documents (e.g. public consultation feedback) |
Running the process — it is already executable
crdm_start and crdm_status were proposed here and should not be built.
The generic workflow tools already run this process, because
processes/process/crdm-requirements.bpmn is a loadable BPMN process like every
other diagram in this repository:
| Instead of | Use |
|---|---|
crdm_start | workflow_start with crdm-requirements and the bean the work hangs on |
crdm_status | workflow_next — what is enabled now, which lane owns it, which skill implements it |
| a phase checklist | workflow_complete, which refuses a step that is not enabled, so phases cannot be claimed out of order |
A second set of tools over the same diagram would be a second answer to
“where are we”, free to disagree with the first. State lives under
beans/workflows/ and is committed, so a sibling session sees the same
position — which is the property a chat-local phase tracker could never have.
Every activity in the agent’s lane carries the skill that implements it, so
workflow_next returns something actionable rather than a step name:
| Step | Skill |
|---|---|
| Detect feature request | crdm-detect |
| Scan / link / ask about the issue | crdm-requirements-workflow |
| Phase 1 — stakeholders, synthesise needs | crdm-requirements-workflow |
| Phase 2 — map the current workflow | bpmn-authoring |
| Phases 3–4 — requirements and impact | crdm-requirements-workflow + content-graph |
| Phase 5 — create beans | todo-manager |
| Phase 6 — implement | prepare-merge |
| Post the round summary | delivery-summary |
| Close the issue | crdm-requirements-workflow |
Three steps also carry a work-plan effect the engine performs: implementing claims the bean, creating beans notes it, and closing resolves it — and resolve only lands once the instance itself has completed, because whether work is done is a judgement.
stakeholder_map — built, but not from the sources proposed
It was specified here as reading “affected roles from harness.config.json and
CODEOWNERS”. Neither exists. This repository has no CODEOWNERS file, and
schemas/harness-config.ts declares no role, owner, maintainer or contact
field. Built as written, the tool would have reported no stakeholders for
every change, forever, and looked like it worked.
It reads what the repository actually has:
- Skills declare
roles:in their front matter — reader, collaborator, owner, auditor. Change a skill and you affect whoever holds its roles. - BPMN lanes name the accountable actor, and each activity names the skill that implements it. So a changed skill reaches a set of lanes across the whole process corpus — editors, the publication manager, clinical SMEs.
That second hop is the one CODEOWNERS could never have given: changing
todo-manager.md reaches the work-plan lane in six separate processes,
which is a far better prompt for “who should hear about this?” than a list
of file owners.
It does not name people, deliberately. crdm-requirements.bpmn documents
its own stakeholder step as “The agent asks the BA who is affected by this
change. The BA knows the domain — the agent does not guess.” The tool
respects that: it enumerates roles, lanes and processes so the BA is asked a
sharper question, and it never outputs a list of humans as “the
stakeholders”. Every report ends with a NOT DETERMINED section naming the
people half, any changed path that maps to no skill (unknown impact, not
absent impact), and any skill whose roles: are undeclared.
bun run cat stakeholder-map <path>... or --since <ref>; stakeholder_map as
an MCP tool.
What is not built yet
This page documents the methodology; the skills and tooling that implement it are in various stages of development. What follows is checked against the repository rather than against intent — a gap list that goes stale is worse than none, because it sends a reader looking for something that is already there, or lets them assume something exists because nobody updated the list.
Built since this page was first written:
- CRDM detection skill —
skills/sdlc/crdm/crdm-detect.mdgives the agent five categories of detection phrasing, an explicit “what is not a feature request” list, and the session-state rules (new session, existing session already in the process, existing session doing content work). The judgement is written down; it is not yet measured, so see the caveat below. stakeholder_map— reports which skills a change touches, the roles they declare, and the process lanes accountable for work that uses them. Built against skills and BPMN lanes rather than the CODEOWNERS andharness.config.jsonfields it was specified from, because neither exists here. It does not name people, because the process says the agent does not guess.- The process is executable —
crdm-requirements.bpmnloads like every other diagram here, soworkflow_start/workflow_next/workflow_completerun it today, andworkflow_completerefuses a step that is not enabled. Every activity in the agent’s lane names the skill that implements it, and three carry the bean operation the engine performs. This is whycrdm_startandcrdm_statusare not on the list below — they would be a second answer to “where are we”, free to disagree with the first. - BPMN diagram of the process itself —
processes/process/crdm-requirements.bpmn, with lanes for the BA / feature requestor, the agent, and stakeholders. - The six-phase workflow as a skill —
skills/sdlc/crdm/crdm-requirements-workflow.md, including the actors table and the issue-association rules.
Not yet implemented:
- Structured requirements templates — no MCP tool scaffolds the CRDM phases into a GitHub issue. The structure described above is written by hand each time.
- Automated impact analysis — the agent can read the content graph and the schema files, but no dedicated tool produces an impact report for a proposed change.
- Review triage tooling — ingesting a Word document with comments and presenting them for structured triage is tracked in #197.
Built and now measured, with a known weakness. The detection skill used
to sit in a third state — “built but unverified”, which is not a milder form
of built. It has since been run against every issue in this repository (27,
the whole population, not a sample) via bun run cat eval:crdm-detect:
| fired | did not | |
|---|---|---|
| is a feature request | 12 | 7 |
| is not | 5 | 3 |
precision 71% · recall 63% · F1 67% — measured 2026-09-18 on main.
Two things that number is not. It runs the skill’s phrase list only, so it is a lower bound on an agent that also applies judgement. And the ground truth is one annotator’s, unblinded — the same agent wrote the labels and the scorer. Fix that before quoting these as a property of the skill rather than of this corpus.
The failure pattern is more useful than the score. The seven misses include #203 itself — the issue that asked for this capability — and #1, the framework design. The five false alarms are two migration records of work already done, two asks to document an existing pipeline, and one bug report.
That points at a specific gap: the skill’s “what is NOT a feature request” list excludes content tasks — writing a section, fixing a typo, reviewing a chapter — but says nothing about records of completed work or documentation about a feature, which are what it actually confuses here. Those two exclusions are the cheapest available improvement.
Tracked in: #203