Running a verification round
How a jurisdiction team invites a panel of practitioners to verify a ready Seed Role and reaches a recorded, exportable consensus.
A verification round is a structured way for invited practitioners to confirm — relationship by relationship — that a Seed Role is right for their jurisdiction. The round emits a signal, never a write: consensus produces an editorial worklist that the authoring team applies in the Role Builder.
This journey is for round owners (the jurisdiction team running the round) and the panelists they invite. It assumes the role has already been authored to ready — see the Seed Roles journey for that.
The round, step by step
One mechanism covers Delphi studies, online workshops and focus groups — the method is a label and a cadence. A round runs through the same seven steps every time.
- 1Open the round — To open a round: go to the jurisdiction dashboard, find the Seed Role's card once it shows the "ready" badge, and click "Start verification round". (The admin Verification rounds page also lists every round in one place.) You then choose the method — Delphi, online workshop or focus group — the close date, and the panel. A round can only open on a role that has cleared authoring.
- 2Invite the panel — The owner invites practitioners by email. Each becomes a lightweight, passwordless panelist account; invited is tracked separately from responded.
- 3Panelists sign in — Each invitee receives a sign-in link and joins without a password. The link is scoped to the round and expires when the round closes.
- 4Cast verdicts — Panelists see a ballot — each profile by title, its items and the IRI/ICI — and cast one verdict per relationship. They can also propose a missing relationship for others to verify.
- 5Owner closes the round — When the window ends the owner closes the round. Closing is what computes the consensus and freezes the result; nothing is final while a round is open.
- 6Apply the worklist — Consensus produces an editorial worklist beside the Role Builder's Edges tab — confirmed, rejected, adjusted and proposed relationships. The authoring team applies it with the normal edge editors.
- 7Optional: a second Delphi round — For a Delphi study, the owner opens round two showing round one's aggregate next to each relationship — the defining Delphi affordance, informed re-rating after the team has acted on the worklist.
The four verdicts
A panelist casts exactly one verdict per relationship. The unit of verification is the relationship — does this profile belong to this role, this item to this profile — not a flat checklist.
- Confirm
- The relationship belongs as shown — the profile, item or information use fits the role as snapshotted for the round.
- Reject
- The relationship does not belong here. Enough rejections turn it into a worklist item to remove.
- Suggest Improvement
- The relationship belongs, but an attribute is wrong. The panelist supplies the corrected value — a different IRI, ICI, classification or relevance.
- Abstain
- The panelist is not qualified to judge this relationship. Abstentions are excluded from the consensus count, so they never dilute a result.
A missing relationship is not a verdict. A panelist who spots one proposes it as a new ballot target, and other panelists then cast verdicts on the proposal in the same round.
How consensus is reached
Consensus is computed by a deterministic function when the round closes — the same inputs always yield the same outcome, with no model in the loop. For each relationship it looks only at the verdicts that were cast (abstentions are set aside) and applies two thresholds: a quorum, a minimum number of responses, and an agreement threshold, the share that must agree. The defaults are five responses and seventy per cent agreement.
A relationship clears as confirmed or rejected when both thresholds are met. If the quorum is not met it is recorded as no consensus — a common, actionable state, not a failure: the owner can extend the window or re-invite. An adjust outcome is a plurality, never a decision: the worklist shows the full spread of suggested values (for example IRI 3 from five panelists, IRI 4 from four) and the authoring team picks.
Privacy and attribution
Panelists are invited and attributed: every verdict is named internally so the owner and a sponsoring body can see who voted which way and read the reasoning. This is what makes the round auditable.
Public attribution is opt-in. By default a published or shared result shows the aggregate only — "verified by 18 panelists" — and a panelist's name appears publicly only if they have opted in. The internal attributed view and the public aggregate view are kept separate.
The verification record
Every closed round is preserved as a verification record — a deterministic export of the whole round: the roster, every relationship and its frozen snapshot, every verdict with its comment and timestamp, and the computed consensus with the full distribution.
An auditable, tamper-evident record
A sponsoring research or professional body can request the full attributed record for a round. The export is deterministic — re-deriving it always produces the same bytes — and a public consumer receives the aggregate view, respecting each panelist's attribution choice.
On close, an immutable copy of the record is archived and its SHA-256 hash is stored. A body can keep the record independently and prove it is unaltered by re-deriving the export and matching the hash — lightweight tamper-evidence, without cryptographic signing.
Before a round can open
A verification round needs a ready Seed Role. Authoring a Seed Role end-to-end is covered in its own journey.
