30  Predictive Validity

30.1 Background: Published studies

Predictive validity of ROAR Foundational Reading Skills (see Section 9.1 for additional information on ROAR Foundational Reading Skills) was first reported by (Gijbels et al. 2024). Gijbels et al. (2024) examined the classification accuracy of ROAR Foundational Reading Skills administered in 1st grade for classifying students who were deemed “at risk” for reading difficulties based on the Fountas and Pinnell (F&P) Benchmark Assessment 8 months later in the fall of 2nd grade. This study included N=130 1st grade students from a public school in California. Students completed ROAR Foundational Reading Skills measures in their classroom and F&P Benchmark Assessments were administered by their classroom teachers. A Generalized Additive Model (GAM) (S. Wood and Wood 2015; S. N. Wood 2017) based on ROAR-Phoneme achieved an AUC=0.70, ROAR-Word achieved and AUC=0.83, and a GAM with ROAR-Phoneme and ROAR-Word achieved an AUC=0.84. The prediction accuracy of ROAR-Phoneme and ROAR-Word for reading skills assessed the following school year with individually-admininstered assessments demonstrated the promise of ROAR as a quick and automated screener.

30.2 Longitudinal studies of grades 1-3

We ran 2 additional studies to assess the predictive validity of ROAR Foundational Reading Skills

  1. Two-year longitudinal study of predictive validity: In a large California school district, all the 1st grade classrooms were administered ROAR Foundational Reading Skills measures three times per year and were followed longitudinally for 2 years. In the fall of 3rd grade, each student was individually administered the Woodcock Johnson Basic Reading Skills (WJ BRS) Composite Index. Based on this criterion measure, we assessed sensitivity and specificity of ROAR at each timepoint for predicting students who were classified as struggling readers with indications of dyslexia. Additionally we report prediction accuracy based on BRS as a continuous measure.
  2. Fall to spring prediction in 1st, 2nd, and 3rd grade: In a second study we assessed predictive validity of ROAR Foundational Reading Skills measures admininstered in the Fall and Winter for predicting individually administered FAST™ earlyReading and FAST™ CBMreading in the Spring (for concurrent validity of ROAR Spring assessment see Chapter 28).

For each study we report Area Under the Curve (AUC), Sensitivity, and Specificity as measures of classification accuracy and Pearson’s \(\rho\) as a measure of prediction accuracy for continuous criterion measures.

30.2.1 Study 1: Two-year longitudinal study with Woodcock Johnson’s Basic Reading Skills (BRS) as the criterion

We implemented our ROAR measures beginning in the first grade. As shown in Table 30.1, ROAR-Word administered in first grade consistently predicts third-grade WJ-BRS outcomes.The correlation between ROAR-Word and WJ-BRS strengthens over time. Table 30.2 demonstrates that ROAR measures can predict reading fluency as early as the first grade, with ROAR-Sentence being the most relevant predictor. The highlighted rows indicate the concurrent validity, where the ROAR measures and WJ-BRS were administered within the same month.

Table 30.1: Predictive validity between ROAR measures and WJ BRS
ROAR Measure ROAR Administration N Correlation
ROAR Word Fall 2021 120 0.685
ROAR Word Spring 2022 120 0.632
ROAR Word Fall 2022 125 0.629
ROAR Word Spring 2023 141 0.738
ROAR Word Fall 2023 165 0.744
ROAR Phoneme Spring 2022 74 0.408
ROAR Phoneme Fall 2022 117 0.539
ROAR Phoneme Spring 2023 133 0.557
ROAR Phoneme Fall 2023 166 0.526
ROAR Sentence Spring 2023 125 0.715
ROAR Sentence Fall 2023 164 0.697
Table 30.2: Predictive validity between ROAR measures and WJ Sentence Fluency
ROAR Measure ROAR Administration N Correlation
ROAR-Word Fall 2021 120 0.703
ROAR-Word Spring 2022 120 0.733
ROAR-Word Fall 2022 125 0.672
ROAR-Word Spring 2023 141 0.713
ROAR-Word Fall 2023 165 0.737
ROAR-Phoneme Spring 2022 74 0.336
ROAR-Phoneme Fall 2022 117 0.486
ROAR-Phoneme Spring 2023 133 0.574
ROAR-Phoneme Fall 2023 166 0.513
ROAR-Sentence Spring 2023 125 0.817
ROAR-Sentence Fall 2023 164 0.831

Based on the WJ-BRS, 32 out of 170 students were identified as high-risk or at-risk struggling readers (scoring below the 50th percentile of the WJ-BRS norms). We treated this classification as the true score. Next, we examined the prediction accuracy of a logistic regression model using ROAR measures taken in the previous year. Figure 30.1 provides further evidence supporting the high sensitivity and specificity of ROAR-Word in predicting dyslexia classification with a lead time of two years.

Figure 30.1: Prediction of Woodcock Johnsons Basic Reading Skills (BRS) risk categories based on a logistic regression model with ROAR measures in previous timepoints.

30.2.2 Study 2: Fall to Spring prediction of FAST™ earlyReading and FAST™ CBMreading

Table 30.3 demonstrates that ROAR-Word in the Fall, among ROAR measures, is the strongest predictor of FAST™ CBMreading performance in the Spring for 1st graders. For 2nd and 3rd graders, both ROAR-Word and ROAR-Sentence are strong predictors of FAST™ CBMreading outcomes. Additionally, Table 30.4 provides further evidence that ROAR-Word in the Fall is a robust predictor of FAST™ earlyReading performance in the Spring. The highlighted rows indicate the concurrent validity, where the ROAR measures and FAST™ were administered within the same month.

Table 30.3: Predictive validity between ROAR measures and FAST™ CBMreading
Grade ROAR Measure ROAR Administration N Correlation
1 ROAR-Word Fall 2023 313 0.725
1 ROAR-Word 2024 Winter 336 0.782
1 ROAR-Word Spring 2024 306 0.777
1 ROAR-Phoneme Fall 2023 305 0.589
1 ROAR-Phoneme 2024 Winter 352 0.647
1 ROAR-Phoneme Spring 2024 332 0.604
1 ROAR-Sentence Fall 2023 263 0.647
1 ROAR-Sentence 2024 Winter 345 0.791
1 ROAR-Sentence Spring 2024 307 0.796
2 ROAR-Word Fall 2023 342 0.748
2 ROAR-Word 2024 Winter 338 0.705
2 ROAR-Word Spring 2024 319 0.678
2 ROAR-Phoneme Fall 2023 350 0.500
2 ROAR-Phoneme 2024 Winter 150 0.404
2 ROAR-Sentence Fall 2023 333 0.765
2 ROAR-Sentence 2024 Winter 330 0.784
2 ROAR-Sentence Spring 2024 322 0.780
3 ROAR-Word Fall 2023 192 0.577
3 ROAR-Word 2024 Winter 163 0.583
3 ROAR-Word Spring 2024 150 0.590
3 ROAR-Phoneme Fall 2023 193 0.363
3 ROAR-Phoneme 2024 Winter 99 0.399
3 ROAR-Sentence Fall 2023 190 0.587
3 ROAR-Sentence 2024 Winter 163 0.600
3 ROAR-Sentence Spring 2024 149 0.594
Table 30.4: Predictive validity between ROAR measures and FAST™ earlyReading
Grade ROAR Measure ROAR Administration N Correlation
1 ROAR-Word Fall 2023 313 0.725
1 ROAR-Word 2024 Winter 335 0.780
1 ROAR-Word Spring 2024 306 0.777
1 ROAR-Phoneme Fall 2023 305 0.589
1 ROAR-Phoneme 2024 Winter 351 0.645
1 ROAR-Phoneme Spring 2024 331 0.601
1 ROAR-Sentence Fall 2023 263 0.647
1 ROAR-Sentence 2024 Winter 345 0.791
1 ROAR-Sentence Spring 2024 307 0.796

We examined the prediction accuracy of a logistic regression model using ROAR measures from Fall 2023 to predict the FAST™ classification (low risk vs. some risk and high risk) in Spring 2024. Figure 30.2 provides evidence supporting the high sensitivity and specificity of ROAR-Word in predicting dyslexia classification in both 1st and 2nd grades. Additionally, ROAR-Phoneme is more useful in 1st grade and ROAR-Sentence proves to be more useful in 2nd grade.

(a) 1st grade prediction of FAST™ CBMreading
(b) 2nd grade prediction of FAST™ CBMreading
(c) 1st grade prediction of FAST™ earlyReading
Figure 30.2: Predictive validity between ROAR measures in the Fall and FAST™ measures in the Spring

We examined the prediction accuracy of a logistic regression model using ROAR measures from Fall 2023 to predict the FAST™ classification (low risk vs. some risk and high risk) in Spring 2024. Figure 30.3 provides evidence supporting the high sensitivity and specificity of ROAR-Word in predicting dyslexia risk classification in 3rd grade.

(a) 3rd grade prediction of FAST™ CBMreading
Figure 30.3: Predictive validity between ROAR measures in the Fall and FAST™ CBMreading measures in the Spring

References

Gijbels, Liesbeth, Amy Burkhardt, Wanjing Anya Ma, and Jason D Yeatman. 2024. “Rapid Online Assessment of Reading and Phonological Awareness (ROAR-PA).” Sci. Rep. 14 (1): 1–16.
Wood, Simon N. 2017. Generalized Additive Models: An Introduction with R, Second Edition. CRC Press.
Wood, Simon, and Maintainer Simon Wood. 2015. “Package ‘Mgcv’.” R Package Version 1 (29): 729.