Six subjects, three raters, one table: work through the ANOVA mean squares and get ICC(1,1), ICC(2,1), ICC(3,1) and the average-measures versions, with the rule for choosing a form and a warning about small samples.
Free sandbox · No credit card · 21 CFR Part 11 aligned
| Rater A | Rater B | Rater C | |
|---|---|---|---|
| Participant 1 | 118 | 121 | 126 |
| Participant 2 | 132 | 135 | 139 |
| Participant 3 | 104 | 106 | 111 |
| Participant 4 | 145 | 142 | 150 |
| Participant 5 | 126 | 129 | 131 |
| Participant 6 | 112 | 116 | 118 |
| Rater mean | 122.8 | 124.8 | 129.2 |
The short answer
The ANOVA
Systolic blood pressure from the hero table (demo data), n = 6 subjects, k = 3 raters, grand mean 125.61 mmHg.
| Source | df | Sum of squares | Mean square |
|---|---|---|---|
| Between subjects (MSR) | 5 | 2,920.94 | 584.19 |
| Between raters (MSC) | 2 | 125.78 | 62.89 |
| Residual (MSE) | 10 | 21.56 | 2.16 |
| Within subjects, raters + residual (MSW) | 12 | 147.33 | 12.28 |
MSW pools the rater and residual sums of squares: (125.78 + 21.56) / 12 = 12.28. The one-way ICC treats rater differences as error; the two-way forms separate them.
The six common forms
| Form (Shrout-Fleiss) | Model | Formula | Value | 95% CI |
|---|---|---|---|---|
| ICC(1,1) | One-way random, single rater | (MSR − MSW) / (MSR + (k−1)MSW) | 0.939 | 0.79 to 0.99 |
| ICC(2,1) | Two-way random, single rater, absolute agreement | (MSR − MSE) / (MSR + (k−1)MSE + k(MSC − MSE)/n) | 0.940 | More complex; use software |
| ICC(3,1) | Two-way mixed, single rater, consistency | (MSR − MSE) / (MSR + (k−1)MSE) | 0.989 | 0.955 to 0.998 |
| ICC(1,k) | One-way, mean of k raters | (MSR − MSW) / MSR | 0.979 | – |
| ICC(2,k) | Two-way random, mean of k | (MSR − MSE) / (MSR + (MSC − MSE)/n) | 0.979 | – |
| ICC(3,k) | Two-way mixed, mean of k | (MSR − MSE) / MSR | 0.996 | 0.984 to 0.999 |
Intervals for the F-based forms use F = MSR / MSE (or MSR / MSW) with the F quantiles for 95% limits (Shrout and Fleiss; McGraw and Wong, 1996). The ICC(3,1) interval comes from F = 271.0 on 5 and 10 degrees of freedom.
Choosing a form
ICC is a family, and reports that say only "ICC" are unusable. Shrout and Fleiss (1979) defined three cases. In case 1 each subject is rated by a different, randomly chosen set of raters (a multi-centre study where whoever is on duty reads the scan), so rater variance cannot be separated. In case 2 the same k raters rate every subject and they are a random sample from a larger pool, so you want to generalise to any similar rater; the coefficient penalises systematic differences between raters. In case 3 the same raters rate everyone and you care only about these raters, for example the three trained nurses at your site; the coefficient ignores a constant offset between raters and measures consistency. The second number in the label is 1 for the reliability of a single rater’s measurement and k for the mean of k raters.
The worked data show why this matters. Rater C reads 4 to 6 mmHg higher than A and B, which is a systematic difference. Consistency ICC(3,1) is 0.989 because the ranking of participants is almost identical across raters. Absolute agreement ICC(2,1) is 0.940 because the offset is counted as disagreement. If the planned measure is "one nurse takes the reading, any nurse", the relevant number is 0.940; if the same rater will measure the participant at every visit, the consistency form may be what you need, and then the right fix for the offset is rater training, not a statistic. McGraw and Wong (1996) and Koo and Li (2016) give clear guidance on matching form to design, and many software packages use different default names, so quote the model, type and definition, as in "two-way random-effects, absolute agreement, single measurement".
The average-measures ICCs are higher because averaging reduces random error. Use them only if the final analysis will really use the mean of k raters or k repeats. Reporting ICC(3,k) = 0.996 for a study in which one rater takes each real-life reading is misleading.
Sample size
One-way ICC, k = 3 raters, assuming the observed ICC is 0.94 or 0.75. The 95% interval from the F distribution (Shrout and Fleiss), rounded to two decimals.
| Subjects | Observed ICC 0.94: 95% CI | Observed ICC 0.75: 95% CI |
|---|---|---|
| 6 | 0.79 to 0.99 | 0.34 to 0.96 |
| 10 | 0.84 to 0.98 | 0.46 to 0.92 |
| 20 | 0.88 to 0.97 | 0.56 to 0.88 |
| 30 | 0.89 to 0.97 | 0.60 to 0.86 |
| 50 | 0.91 to 0.96 | 0.64 to 0.84 |
The width, not the point estimate, drives the plan. A planning formula for a target interval width or a minimum acceptable ICC is given by Walter, Eliasziw and Donner (1998) and Bonett (2002); ask a statistician to apply it to your design.
Build the eCRFs, edit checks and visit schedule in the free sandbox with every feature. No credit card, no time limit. You pay only when you go live with real participants.
Interpretation and caveats
Koo and Li (2016) suggest describing ICC below 0.5 as poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good and above 0.9 excellent, and they stress that the confidence interval, not the point estimate, should be used to classify. These are rules of thumb, not standards: a reliability of 0.8 may be fine for a research measure and inadequate for a diagnostic cut-off. In the worked example the lower bound of the one-way interval is 0.79, which sits in the "good" range even though the point estimate is "excellent". A statistician should set the acceptable level in advance from the use of the measurement.
ICC depends on the spread of the sample. If every participant has a blood pressure between 118 and 124, between-subject variance is small and the ICC is low even when raters are within 2 mmHg of each other. Conversely, a sample with a wide range inflates the ICC. For clinical decisions, the standard error of measurement (the square root of the error variance, about 1.47 mmHg for MSE = 2.16 in the example) and the Bland–Altman limits of agreement are more informative in the original units. ICC also assumes roughly normal, homoscedastic errors; if error grows with the size of the measurement, a log transform or a different method is needed. For categorical ratings, ICC is the wrong tool: use the Cohen’s kappa calculator.
Test-retest reliability uses the same logic with occasions in place of raters, and repeated measures that are meant to differ over time (a drug effect between visits) are not reliability data at all. ICC is also the quantity behind clustering: the design effect used in the cluster-randomized sample size calculator and the prevalence sample size calculator is 1 + (m − 1)ρ, where ρ is an intraclass correlation. Do not borrow a value from a reliability study for a clustered trial without checking that the measure and the unit of clustering match.
Design the reliability study so the table is interpretable. Use a single written measurement procedure, make the order of subjects and raters random, keep each rater unaware of the others’ readings, and fix the interval between repeated measurements so that true change is negligible but memory of the previous reading is not an issue. Use enough raters to represent those who will use the measurement in practice; three trained raters tell you little about a pool of fifty. Report the ICC form, the 95% interval, the number of subjects and raters, the standard error of measurement and the sample range together. If a few subjects drive the result, show the data. As with every calculator on this site, a statistician should choose the model and check the assumptions before the number goes into a protocol or a paper.
Collecting reliability data
A reliability study needs a clean subjects-by-raters table. Build one eCRF form for the measurement with a numeric field and unit, a single-choice rater identifier and the measurement date and time, or use a table section for repeating rows when a participant is measured several times in one visit. In Capture, range high and range low edit checks raise an auto-query when a reading is implausible, calculated fields can derive the mean of repeated readings read-only, and visit windows keep test-retest intervals consistent. The field-level audit trail records user, timestamp, old value, new value and reason, so an edit after the fact is visible. Exports in CSV or Excel are wide format, one row per submission, with a data dictionary, which reshapes simply to the subjects-by-raters layout the ANOVA needs.
This pattern suits device and biomarker validation studies (see digital biomarker validation study software and EDC for wearable device studies), medical-device studies (EDC for medical device clinical trials) and diagnostic reader studies (EDC for diagnostic accuracy studies). The measurement itself can start from the vital signs eCRF template. The ICC is computed in your statistical software from the export; Capture provides the structured, audit-trailed inputs, and does not itself calculate the coefficient.
Decision rule
One-way model, ICC(1,1) for a single measurement.
Two-way random, absolute agreement: ICC(2,1). The default when raters can differ in practice.
Two-way mixed, consistency: ICC(3,1), and check the rater means for an offset.
Report the average-measures form and say k.
Use kappa or weighted kappa instead.
State the model, type (agreement or consistency), single or average, n, k, the 95% CI and the software.
It is the proportion of total variance in measurements that is due to true differences between subjects rather than rater disagreement and error. It runs from 0 (no reliability) to 1 (perfect), and is used for continuous measurements read by several raters or repeated over time.
Both use the same raters for all subjects. ICC(2,1) treats the raters as a random sample and penalises systematic differences between them (absolute agreement). ICC(3,1) treats the raters as fixed and ignores a constant offset (consistency). In the worked example they are 0.940 and 0.989.
Run a two-way ANOVA without interaction to get MSR, MSC and MSE. Then ICC(3,1) = (MSR − MSE) / (MSR + (k−1)MSE), and ICC(2,1) = (MSR − MSE) / (MSR + (k−1)MSE + k(MSC − MSE)/n). For the example, MSR 584.19, MSC 62.89, MSE 2.16, n = 6, k = 3.
A frequently quoted guide is below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good and above 0.9 excellent, applied to the confidence interval rather than the point estimate. These are conventions; the acceptable level depends on how the measurement is used.
It depends on the expected ICC, the number of raters and the interval width you want. With 3 raters and an observed ICC of 0.94, six subjects give an interval of 0.79 to 0.99 and thirty give 0.89 to 0.97. Use a planning method such as Walter, Eliasziw and Donner and confirm with a statistician.
Use single-measure ICC if the real measurement will be one rater’s reading, and average-measures ICC only if the final analysis will use the mean of k readings. Average measures are always higher, so quoting them for a single-reading use overstates reliability.
No. For categorical or ordinal ratings use Cohen’s kappa or weighted kappa. ICC assumes a continuous, approximately normal measurement. See the Cohen’s kappa calculator page for a worked 2x2 example.
No. Capture collects each rater’s measurement as audit-trailed eCRF data with edit checks and exports it to CSV or Excel with a data dictionary, and the ICC is computed in your statistical software. This page is a reference aid, not validated software.
Keep exploring
Cohen’s kappa calculator
Agreement for categorical ratings.
Digital biomarker validation study software
Reliability and validity data collection.
EDC for wearable device studies
Repeated readings from devices.
Cluster-randomized sample size calculator
Where ICC enters the design effect.
Two-means sample size calculator
Size a trial on a measured outcome.
Vital signs eCRF template
Start the measurement form.
Build the eCRFs, edit checks and visit schedule in the free sandbox with every feature. No credit card, no time limit. You pay only when you go live with real participants.