3  ROAR Scores and Norms

ROAR has been developed through a collaborative, co-design process with schools around the United States. The development, validation, score-reporting, and norming samples reflect the contributions of hundreds of schools that have collaborated with the ROAR team through a Research Practice Partnership (RPP) model (Laura Wentworth et al. 2023; L. Wentworth et al. 2021). The goal of this model is to make sure that the diverse interests of stakeholders (teachers, students, parents, school administrators, etc.) representing the incredible diversity in the U.S. education system have a voice in guiding the research and development process of the tools used in their schools. Section 3.2 shows the distribution of ROAR partner schools around the United States.

3.1 ROAR Scores

The ROAR assessment score reports include different types of scores that each have intended use cases. The ROAR Next Steps Guide provides detailed descriptions of how to interpret scores and use them to guide instruction and/or intervention. The ROAR Family Guide provides an explanation of scores and a guide for support at home, designed for parents and guardians.

One important consideration for interpreting scores on any assessment is participant effort, concentration, and engagement. A score is only an accurate representative of the participant’s ability level if the participant is engaged and tries their hardest even as the assessment gets difficult. For assessments that are individually administered (e.g. by a teacher), the administrator might get a qualitative impression of the participant’s effort and focus. If the same person is administering the assessment and interpreting the scores (e.g. classroom teacher or reading specialist), this qualitative impression can be helpful. However it can also be a source of bias and it is hard to standardize the criteria for judging engagement. For automated, online assessments like ROAR, participant disengagement might be a particular concern and needs to be considered when interpreting scores. Each ROAR measure has a defined criteria, grounded in research, for identifying disengaged participants and flagging unreliable scores (for example see Section 15.2 and Section 16.2). This criteria is defined in an algorithm that takes into account a) the response time distribution and b) pattern of responses on the assessment. Scores for any participant that are flagged for disengagement or other issues that might affect the interpretation of the score are flagged in the ROAR Score Report. Figure 3.3 shows an example ROAR Score Table indicating which students need support in Phonological Awareness, Single Word Reading and Sentence Reading Efficiency and disengaged participants with unreliable data are flagged with grayed out scores (dialog box provides additional information).

Figure 3.1: Understanding ROAR scores using the normal curve (Gaussian distribution).

Types of Scores

• Raw Scores and Scaled Scores: A raw score is the basic measure of a student’s performance on the test. A scaled score puts the raw score on an interval scale to make it more interpretable. Each assessment reports a raw score and the scoring rules for that raw score are detailed in the introduction to that assessment (for example, see Section 5.2 for the scoring rules for ROAR-Word). Most of the assessments use item response theory (IRT; see Section 4.2) and computer adaptive testing (CAT; see Chapter 4) for scoring, though some timed assessments like ROAR-Sentence (see Section 6.4) use other types of scoring models. All assessments that use an IRT model for scoring report Scaled Scores. Scaled Scores are comparable across grades and over time and are useful for tracking growth in reading skills (e.g., response to intervention (RTI)).

• Percentile Scores: The percentile refers to a student’s rank within their grade level on the given skill. The percentile is the number of students out of 100 who have lower scores. Percentile scores are computed by comparing raw scores to a norming table. A norming table captures the distribution of scores for each age bin in a lookup table providing the percentiles associated with each raw score. Percentile Scores are useful for identifying students who are struggling relative to their peers (or relative to national norms). Figure 3.4 (a) shows a screenshot from ROAR Score Reports displaying Percentile Scores. The norming table for percentile scores are computed in 2 ways:

  1. Based on ROAR Norms. Section 3.2 shows the participating ROAR schools, Section 3.3 presents school characteristics and Section 3.4 show student characteristics.
  2. Based on linking ROAR scores to criterion measures like the Woodcock Johnson Basic Reading Skills (WJ BRS) Standard Scores. This linking allows ROAR-Word scores to be interpreted with direct reference to the criterion measure that is often used to define dyslexia risk (for example see Section 23.1).

• Standard Scores: A standard score is a way of showing how performance compares to other kids of the same age or grade. The standard score is comparable within a grade level, but not across grade levels or over time. Figure 3.4 (b) shows a screenshot from ROAR Score Reports displaying Standard Scores. Age standardized scores for ROAR-Word put scores for each age bin on a standard scale (normal distribution, \(\mu=100\), \(\sigma=15\), see Figure 3.1) and are computed in 2 ways:

  1. Based on ROAR Norms
  2. Based on linking ROAR scores to criterion measures such as WJ BRS Standard Scores in the case of ROAR-Word (Section 23.1.1) and CTOPP Standard Scores in the case of ROAR-Phoneme (Chapter 25).

• Support Categories: For each measure, ROAR recommends students who are in need of extra support. Support categories can also be interpreted as indicating risk of reading difficulties such as dyslexia (for more information see Chapter 28 and Chapter 9). Dyslexia refers to the lower end of a continuum of reading skills and there is no agreed upon cutoff. The 25th percentile based on national norms is a common cutpoint that is used to indicate students who are in need of additional support and that is the cut point that is implemented in ROAR Support Categories. Students below the 25th percentile are recommended for additional support. Students between the 25th and 50th percentile are indicated that the skill is still developing. Students above the 50th percentile are indicated as being at grade level (achieved skill). For example, Figure 3.2 shows the distribution of support categories from a hypothetical school district in the ROAR Score Report and Figure 3.3 shows a ROAR Score Table indicating which participants need support in which skills.

Figure 3.2: Screen shot from ROAR Score Reports showing the distribution of students for a hypothetical school district that Need Extra Support (red; below 25th percentile), Developing Skill (yellow; 25th - 50th percentile), Achieved Skill at grade level (green; above 50th percentile)

Figure 3.3: Screen shot from ROAR Score Reports showing support categories for individual participants from a hypothetical school district. Red indicates students who Need Extra Support (below 25th percentile), yellow indicates students still Developing Skill (25th - 50th percentile), and green indicates students that Achieved Skill at grade level (above 50th percentile). Grayed out scores indicate scores that were flagged as unreliable. Mouse hover-over opens a box with additional information on the score explaining why it was flagged to provide teachers additional context.

(a) ROAR Score Report displaying Percentile Scores on Risk Categories

(b) ROAR Score Report displaying Standard Scores on Risk Categories

Figure 3.4: ROAR Score Reports can toggle between different display formats, overlaying Percentile Scores or Standard Scores on Risk Categories.

3.2 Map of ROAR schools

ROAR is being used by schools across the United States. The vast majority are in California. See the map of participating in ROAR in each state. ROAR is in the Southwest, South, Northeast, and Midwest, and ROAR will be in the Northwest in Fall 2024.

Figure 3.5: Map of collaborating ROAR schools in the United States.

3.3 Table of school characteristics

ROAR is designed to serve schools nationwide. Through our research-practice partnership model, we seek to work with a broad scope of schools and community organizations representing the diversity of organizations across the country. Table 3.1 describes the schools and student populations in our norming sample.

	Overall
	(N=108)
School Level
Elementary	63 (58.3%)
Secondary	33 (30.6%)
Combined	12 (11.1%)
School Type
Public	58 (53.7%)
Charter	40 (37.0%)
Private	10 (9.3%)
Locale
City	68 (63.0%)
Rural	10 (9.3%)
Suburb	26 (24.1%)
Town	4 (3.7%)
% Free or Reduced Meals
Mean (SD)	57.9 (33.5)
Median [Min, Max]	71.0 [0, 99.4]
Missing	25 (23.1%)
% Hispanic
Mean (SD)	47.9 (31.9)
Median [Min, Max]	54.5 [0, 97.7]
% American Indian/Alaskan Native
Mean (SD)	0.244 (0.538)
Median [Min, Max]	0 [0, 3.12]
% Asian and Pacific Islander
Mean (SD)	11.0 (14.5)
Median [Min, Max]	3.61 [0, 54.6]
% Black
Mean (SD)	14.5 (22.7)
Median [Min, Max]	3.17 [0, 98.2]
% White
Mean (SD)	20.2 (25.4)
Median [Min, Max]	7.91 [0, 97.4]
% Multi-Racial
Mean (SD)	5.60 (6.41)
Median [Min, Max]	2.60 [0, 28.6]

Table 3.1: School Characteristics and Demographics

3.4 Table of student demographics where individual demographics are available

For several analyses throughout the ROAR Technical Manual, we take into account individual student demographics. For these analyses, we rely on a group of students for whom we have individual demographic data. Table 3.2 shows descriptive statistics for the group of students for whom we currently have individual demographics. This group represents about half of our overall sample. While many of our partnering schools share demographic data with us, some do not. Others share race/ethnicity but not other important demographics such as home language, English learner status, and indicators of socioeconomic status such as free and reduced meal eligibility. Depending on the analysis, we will use a subset of this data so that we can achieve a representative sample. Where applicable, this is clarified throughout the ROAR Technical Manual, and updated sample demographic tables are provided.

	N	%	% Missing
Female	4617	42.91	11.92
Free or Reduced Lunch	1143	10.62	48.87
English Language Learner	1792	16.66	40.86
Special Education	381	3.54	43.94
Race/Ethnicity
Hispanic Ethnicity	3194	29.69	0.00
White	3713	34.51	0.00
Black or African American	693	6.44	0.00
Asian	1593	14.81	0.00
American Indian or Alaska Native	117	1.09	0.00
Hawaiian or Other Pacific Islander	51	0.47	0.00
Multiracial	976	9.07	0.00
Total	10759

Table 3.2: Individual Student Demographics

References

Wentworth, Laura, Paula Arce-Trigatti, Carrie Conaway, and Samantha Shewchuk. 2023. Brokering in Education Research-Practice Partnerships: A Guide for Education Professionals and Researchers. Taylor & Francis.

Wentworth, L, R Khanna, M Nayfack, and D Schwartz. 2021. “Closing the Research-Practice Gap in Education.” Stanford Social Innovation Review 19 (2): 57–58.