18  Reliability of ROAR-Phoneme

18.1 Background: Published studies

Gijbels, Burkhardt, and Ma (2024) reported high correlations between ROAR-Phoneme and standardized measures of PA (CTOPP-2, r=.80) for children from Pre-K through fourth grade and exceptional reliability for the ROAR-Phoneme composite score (\(\alpha=0.96\)). Each individual subtest was also highly reliable:

  • First Sound Matching (FSM) \(\alpha=0.89\)
  • Last Sound Matching (LSM) \(\alpha=0.92\)
  • Deletion (DEL) \(\alpha=0.85\)

Note: These reliability estimates are based on a larger item pool than those in the operational IRT scaling.

18.2 Reliability by subtest

Results from the Rasch scaling were used to compute empirical reliability (\(\rho_{xx^\prime}\)). It is computed based on the estimated variance of \(\hat{\theta}\) relative to the estimated standard error (\(\widehat{SE}(\hat{\theta})^2\)) using Equation 24.1.

Model Empirical Reliability 95% Confidence Interval N
Deletion (DEL) Model 0.76 0.76 to 0.76 5645
First Sound Matching (FSM) Model 0.78 0.77 to 0.78 5645
Last Sound Matching (LSM) Model 0.74 0.74 to 0.74 5645
Combined Model 0.89 0.88 to 0.89 5645
Table 18.1: Empirical reliability of ROAR-Phoneme subtest and composite models.

18.3 Reliability of ROAR-Phoneme composite across demographic groups

The subgroup tables below use thetas from the combined (composite) model. Students who answered every administered item correctly were excluded from the reliability calculation, since their theta and SE are determined by the prior rather than by the data. After this filter, 5645 of 5645 calibration students remained (0 excluded). Overall composite empirical reliability was 0.89 (95% CI: 0.88 to 0.89).

Grade N Empirical Reliability 95% CI
All 5645 0.89 0.88 to 0.89
Kindergarten 867 0.80 0.79 to 0.82
1 1674 0.87 0.86 to 0.87
2 1174 0.88 0.87 to 0.88
3 665 0.85 0.84 to 0.86
4 340 0.84 0.83 to 0.86
5 329 0.84 0.82 to 0.85
6 596 0.87 0.86 to 0.88
Table 18.2: Reliability of ROAR-Phoneme by grade.

References

Gijbels, Liesbeth, Amy Burkhardt, and Jason D Ma Wanjing Anya and Yeatman. 2024. “Rapid Online Assessment of Reading and Phonological Awareness (ROAR-PA).” Scientific Reports 14 (1): 1–16.