Background: Published studies
Bhat et al. (2024) reported marginal reliability of 0.85 for ROAR-Fonema (see more detail in Section 21.1).
Reliability of fixed-length ROAR-Fonema
Table 23.1 reports marginal reliability computed based on data from 4128 students under the IRT model for 2 different versions of ROAR-Fonema: the short, 20 item version and the long 54 item version. The computer-adaptive ROAR-Fonema is planned for release in fall 2024 and will be more efficient and reliable with fewer items. Reliability (\(\rho_{xx^\prime}\)) is computed based on the estimated variance of \(\hat{\theta}\) relative to the estimated standard error (\(\widehat{SE}(\hat{\theta})^2\)) using Equation 23.1:
\[
\hat{\rho}_{xx^\prime} = \frac{\widehat{VAR}(\hat{\theta})}{\widehat{VAR}(\hat{\theta}) + \widehat{SE}(\hat{\theta})^2},
\tag{23.1}\]
To ensure that ROAR-Fonema is fair and equitable for different demographic groups, we also report reliability by gender (Table 23.2), eligibility for free and reduced price lunch (Table 23.3), English learner status based on state of California designations (Table 23.4), primary langauge spoken (Table 23.5), special education (Table 23.6), ethnicity (Table 23.7), and race (Table 23.8).
References
Bhat, Kruttika G., Alexa Mogan, Ana Saavedra, Mia Fuentes-Jimenez, Julian M. Siebert, Wanjing Anya Ma, Carrie Townley-Flores, et al. 2024.
“Shared and Unique Influences of Phonological Processing on Reading and Math.” OSF Preprints.
https://doi.org/10.31219/osf.io/em3bg.