Validating an AI device means proving that one fixed version of an algorithm performs on a defined population against an agreed ground truth, and that readers using it do no worse. Capture does not run your model. It records the cases, model version, reader sessions, ground truth and subgroup data with one audit trail. Build it free in the sandbox.
Free sandbox · No credit card · 21 CFR Part 11 aligned
| Session 1 | Washout | Session 2 | |
|---|---|---|---|
| Readers 1 and 2 | Unaided, cases A | 4 weeks | With AI, cases B |
| Readers 3 and 4 | With AI, cases A | 4 weeks | Unaided, cases B |
| Recorded per read | Reader ID, case ID, mode | Date | Finding, confidence, time |
What an AI device validation study needs from its data capture
The study design
An AI-enabled device is validated by showing that a specific, fixed version of its software gives accurate and reliable output in its intended population and setting. Two designs are common. A standalone performance study compares the algorithm’s output to a reference standard and reports measures such as sensitivity and specificity. A reader study compares clinicians with and without the device, often with several readers each reading many cases (a multi-reader multi-case design), usually with a washout interval between sessions. Studies may be retrospective on archived cases, prospective in routine care, or both. Which one you run changes what the data capture must hold, so settle it in the protocol before building forms.
The model version is the field that distinguishes this from a general diagnostic accuracy study. An algorithm that is updated mid-study is a different device. Record the model name, version and build identifier against each case or output, and add an edit check that raises a query when a version outside the declared locked version is entered. Regulators have also described change control plans for devices that will be updated after authorisation; whether one applies is a regulatory question for your team, and any change within a study should be a documented protocol amendment, not a quiet update. See ecrf versioning for mid-study changes for how form changes are handled.
Capture is the study database and does not host the algorithm, images or DICOM files. Keep images in your image archive or reader platform, and store a pseudonymised case ID and the acquisition metadata in the eCRF so the two can be joined. The device output, the reader’s finding and the ground truth are entered by the people who hold them, and the exported dataset joins them by case ID. This division also keeps personal data out of the study database where the protocol allows, which supports your GDPR obligations.
A reference standard may be a biopsy result, a follow-up outcome, a consensus of expert readers or a laboratory assay. Record the type, the evidence date, the reader or panel, and whether the reader was blinded to the device output. If an adjudication panel resolves disagreement, keep each panelist’s call and the final decision on separate rows. Cases where no ground truth could be established should be counted with a reason, because the way indeterminate cases are handled can change performance measures.
Case and model
Case ID (pseudonymised)
CASEIDAcquisition device
ACQDEVModel name and version
MDLVERDevice output
AIRESDevice score
AISCOREGround truth
GTRESAdjudicated by panel, 2 of 3
Protocol to build
| Protocol element | What the data looks like | Where it lives in Capture |
|---|---|---|
| Case eligibility | Source, inclusion and exclusion criteria, intended-use fields | Eligibility form with edit checks and skip logic |
| Case characteristics | Age, sex, site, acquisition device, condition prevalence strata | Demographics form plus a case metadata form |
| Model version and output | Name, version, build, output, score, run date | Case eCRF form with a version edit check |
| Reader reads | Reader ID, case ID, mode (aided or unaided), session, finding, confidence | Repeating-row reader form; reader as a field |
| Ground truth | Reference type, result, evidence date, panel, adjudication | Site and adjudicator eCRF forms |
| Safety and device deficiency | Device deficiency, unexpected output, patient impact if prospective | Adverse event form and SAE report form |
| Data cleaning and lock | Queries on missing pairs, version breaks, audit trail | Query management and database lock |
Capture records the data your protocol defines. It does not run, host or validate the algorithm, and it does not decide performance.
Upload the protocol, let the AI draft the visits and forms for review, then enter ten practice cases with a deliberate version mismatch and a missing ground truth. Free sandbox.
Quality and traceability
Reader studies are vulnerable to order effects, memory of earlier reads and unequal case difficulty. Record the session, the date, the reader and the mode on each read, and add an edit check that raises a query if a reader appears twice on the same case in the same mode. If your design requires a washout, a check on the days between sessions makes a short washout visible to a monitor. Reader experience (years, subspecialty) is a baseline field on a reader form, so subgroup analyses by experience are possible.
Blinding matters on the ground-truth side too. The person who establishes the reference result should not see the device output if the protocol says so. In Capture, blinded roles never receive treatment-arm values, which is enforced in masked database views and row-level security. Use separate forms and roles for device output, reader reads and ground truth, and review the role-based access page. When a result is changed after the fact, for instance a re-read, the field-level audit trail shows who, when, the old and new value and the reason. This is the kind of provenance a regulatory reviewer will ask for. The audit trail software page describes how it works.
Prospective studies in routine care add the usual clinical trial duties. Eligible patients are consented (eConsent), device-related events and deficiencies are recorded and escalated, and the study may sit under a device standard such as ISO 14155; see ISO 14155 clinical investigation software and EDC for medical device clinical trials. If your device is a digital therapeutic rather than a diagnostic or triage tool, see clinical validation software for digital therapeutics. The regulatory framework for AI in the EU is described on the EU AI Act and clinical trial software page; this page does not give regulatory advice.
Build sequence
A realistic order of work for a small manufacturer or an academic validation team.
Upload the protocol (PDF, DOCX or DOC) and let the AI study builder propose the schedule and forms. Nothing is saved until a person reviews it.
Pseudonymised case ID, case metadata, locked model version, output and score, with a version edit check.
Repeating reader rows with mode and session, and ground-truth and adjudication forms with separate roles.
Strata as baseline fields and a test export that the statistician opens in their package.
Run practice cases through a version mismatch, a re-read and a missing ground truth, approve the forms and start the paid live phase when real data entry begins.
Sites and exports
Validation sets often come from several hospitals so that the device is tested across scanners and populations. Capture supports site-level participant or case numbering, a site coordinator portal and by-site filtering and exports; see multi-site clinical trial management. Source data verification can be set per field, so monitors can focus on ground truth and eligibility (risk-based monitoring and SDV).
Exports come as CSV or Excel with an automatic data dictionary, one row per submission, and CDISC SDTM export gives SAS XPT datasets with Define-XML (SDTM export, export EDC data to R, SAS and SPSS). For the sample size of a paired accuracy study, see the power analysis calculator and the non-inferiority sample size calculator as starting points; a statistician should confirm the design.
Before first case
Name, version and build recorded in the protocol and as a controlled value in the form.
Reference type, who establishes it, blinding and adjudication rules.
The threshold used for positive and negative output, written before data are seen.
Readers, sessions, order, washout and the case split.
Site, acquisition device, sex, age and any other strata listed as baseline fields.
Where images live, and how pseudonymised case IDs link them to the study record.
No. Capture is the study database. It records cases, model version, device output, reader reads and ground truth, and it exports the joined dataset. Algorithms and images stay in your own systems.
Record the model name, version and build on each case and add an edit check that raises a query when a value other than the declared locked version is entered. A real model change should go through a documented amendment.
Yes. A repeating-row form can hold reader ID, case ID, mode, session, date, finding and confidence, with checks for duplicate reads and short washouts.
The core idea is similar, but AI validation adds model version control, aided and unaided reader sessions, subgroup strata and a clear line between algorithm output and clinician reads. See the diagnostic accuracy page for the index-test and reference-standard basics.
No. It captures and exports study data with an audit trail. Regulatory strategy, submissions and obligations under device and AI rules are separate work, and this page is not regulatory advice.
Capture provides 21 CFR Part 11-aligned controls: a field-level audit trail, electronic signatures and role-based access. Compliance is shared with the sponsor’s own validated use of the system. This is not legal advice.
Keep exploring
EDC for diagnostic accuracy studies
Index test, reference standard and reader basics.
EDC for medical device clinical trials
Device studies, deficiencies and ISO 14155.
EU AI Act and clinical trial software
What the EU AI Act means for trial software.
Digital therapeutics validation software
Validation of software as a treatment.
Capture eCRF builder
Design forms, edit checks and visits.
Free sandbox with every feature. No credit card, and you pay only when you go live with real participants.