3  Administration Procedures for ROAR Comprehension Suite

All assessments in the ROAR Comprehension Suite are designed to be administered in a whole class setting. They are administered entirely through an online platform and have been tested across devices (e.g., Chromebooks, iPads, etc.) and web browsers (e.g., Firebox, Safari, Chrome, and other Chromium browsers) that are commonly used in US schools. ROAR is a browser-based app that can be launched through common learning management systems such as Clever and ClassLink or by logging in at https://roar.education.

3.1 Student Experience

When students log in to the ROAR dashboard, they are presented with a series of short, gamified assessments that are appropriate for their grade level. When students click on an assessment icon, the assessment opens and students are provided with instructions that they can hear through headphones. All assessments are narrated by engaging characters that walk students through the instructions and guide them through practice to ensure understanding. Students respond to the assessment items using the keyboard, mouse, trackpad, or touch screen capabilities of the computer or tablet. Once a given assessment is complete, the students are returned to the dashboard where the next assessment is available. Once all of the assessments on the student dashboard are completed, students receive a message that they have finished the assessment and can log off.

3.2 Assessment Timing and Efficiency

The ROAR Comprehension Suite is designed for efficient administration, with most assessments taking 5-10 minutes to complete (see Table 2.1). A complete administration of all four assessments typically takes 25-35 minutes, making it feasible to complete in a single class period or across multiple shorter sessions, depending on school scheduling needs.

Criteria for identifying disengaged participants and flagging unreliable scores: ROAR assessments are designed to be totally automated, making it possible to efficiently assess a whole district simultaneously. A concern about automated assessments is that without individual administration and monitoring, some students might disengage and provide data that is not representative of their true ability.

Criteria for flagging unreliable scores: Participants with rapid responding patterns combined with low accuracy are flagged as unreliable scores in ROAR score reports and are excluded from analyses since scores do not accurately represent the participant’s ability (see example from the ROAR technical manual for Foundational Reading skills). Teachers can choose whether to re-administer ROAR or interpret data cautiously in relation to other data sources and contextual factors.

3.3 Test Administrator Responsibilities

Test administrators are responsible for supporting students in logging in and staying focused and on task. They are not responsible for administering the items directly, as all instruction and item presentation is automated. Test administrators or proctors should not interfere with students’ activity during the short assessments, though it is appropriate to answer clarifying questions about the instructions.

3.3.1 Qualifications and Training

Test administrators do not need to hold specific qualifications, but they should have dedicated time to familiarize themselves with the appropriate guidance materials:

  • ROAR Teacher Guide for certified teachers

  • ROAR Quick-Start Guide for Paraprofessionals and Classroom Assistants for support staff

  • ROAR Guidance for Students Receiving Special Education Services and 504 Accommodations for students with exceptionalities

  • ROAR Guidance for English Language Learners for multilingual students

3.3.2 Key Administrator Functions

Administrator responsibilities encompass several critical areas that ensure valid and reliable assessment administration:

  • Technical support involves assisting with login procedures and basic technical troubleshooting, helping students navigate any platform-related issues that might arise during testing. 

  • Environmental management requires maintaining appropriate testing conditions with minimal distractions and a quiet atmosphere conducive to focused work. 

  • Instructional clarification allows administrators to answer questions about general directions without providing item-specific help that might compromise assessment integrity.

  • Ongoing monitoring involves observing student engagement and flagging any technical or behavioral issues that could affect score validity. 

  • Finally, accommodation support requires implementing approved accommodations while maintaining assessment validity, ensuring that students with documented needs receive appropriate support without compromising the measurement properties of the assessments.

3.4 Assessment-Specific Considerations

3.4.1 Oral Language Assessment (Syntax)

This assessment requires students to listen to audio prompts through headphones or speakers. Administrators should ensure all students can hear the audio clearly and that volume levels are appropriate. Students select responses by clicking on pictures, making these assessments accessible to students with limited reading skills.

3.4.2 Written Language Assessments (Written Vocabulary, Morphology, Inferencing)

These assessments require students to read text independently. Administrators should ensure adequate lighting and that students are positioned to see screens clearly. Students should be encouraged to take their time reading passages and questions carefully. Browser zoom should be set at 100% for optimal formatting.

3.5 Accommodations and Accessibility

3.5.1 Students with Disabilities

For students receiving special education services or those with 504 plans, administrators should review accommodation guidelines to ensure appropriate support while maintaining score validity and reliability. Assistive technology such as screen readers and other devices compatible with web browsers can be utilized when appropriate. Environmental modifications, including separate testing space, preferential seating, or reduced distractions, may be implemented based on student needs. Presentation modifications such as larger text size or high contrast display settings can be adjusted through browser accessibility features though changing the browser zoom can affect the formatting of the assessment so should be used carefully.

3.5.2 English Language Learners

English language learners should be assessed using standard procedures, as the assessments are designed to measure English language proficiency in the targeted domains. However, administrators should ensure students understand general testing procedures and may provide clarification of directions in the student’s first language if needed. Administrators should consider that oral language assessments (Picture Vocabulary, Syntax) versus written language assessments provide different information about the student’s developing English proficiency. Results should be interpreted in the context of the student’s English language development stage, recognizing that performance may reflect both the targeted skill and overall English proficiency.

Currently, assessments are available in English only. Administrators should consult their state or district guidelines regarding appropriate assessment participation for English learners at different stages of English language development (for example, California provides an ELA exemption for newcomers enrolled in U.S. schools for less than 12 months). These guidelines should inform decisions about assessment administration timing and interpretation.

3.6 Technical Requirements and Setup

3.6.1 Device Requirements

Successful administration requires several technical components. A stable broadband internet connection is recommended for optimal performance. Current versions of Chrome, Firefox, or Safari browsers should be used to ensure compatibility. Audio capability through headphones or speakers (if testing individually) is essential for narrated instructions and oral language assessments. Input devices including mouse, trackpad, touchscreen, or keyboard enable student responses across different device types. A minimum 9-inch screen size is recommended for optimal display of assessment materials.

3.6.2 Network Considerations

Assessment administration involves several network considerations that schools should address prior to testing. The assessments are designed to work with typical school internet connections, though adequate bandwidth should be ensured for simultaneous whole-class administration. School firewalls should be configured to ensure the roar.education domain is accessible without restrictions. The system is designed to handle thousands of users simultaneously within a single classroom or across multiple classrooms. Assessments require continuous internet connection throughout administration, as offline capability is not currently available.

3.7 Quality Assurance During Administration

3.7.1 Monitoring Student Engagement

Administrators should be alert to signs of disengagement that might compromise score validity. Rapid responding, where students click through items without reading, may indicate disengagement or technical issues (though this behavior can be detected and is flagged in the score reports). Off-task behavior, such as students being distracted by other activities, should be addressed promptly. Technical difficulties, including problems with audio, display, or input devices, require immediate attention to ensure valid assessment conditions. Fatigue in students, particularly during longer sessions, should be monitored and addressed through appropriate pacing or breaks if needed. In these situations where the integrity of the assessment is compromised the student should be re-administered ROAR in a focused environment.

3.7.2 Ensuring Reliable Data

Several factors contribute to reliable data collection during assessment administration. The testing environment should be maintained as quiet and focused, minimizing distractions that could interfere with student performance. Clear instructions ensure that all students hear and understand initial directions before beginning the assessment (please refer to the Teacher Administration Guide for more detailed information.) 

3.7.3 Flagging Unreliable Scores

The ROAR system automatically flags potentially unreliable scores based on response patterns such as extremely rapid responses with low accuracy. Administrators should review flagged scores in consultation with assessment coordinators to determine appropriate next steps. Readministration may be considered for flagged scores if circumstances warrant and conditions can be improved. Any unusual circumstances that might affect score interpretation should be documented for future reference. Flagged scores should be used cautiously in educational decision-making, with consideration for other assessment data and contextual factors.

3.8 Post-Administration Procedures

3.8.1 Immediate Actions

Several immediate actions help ensure successful assessment completion and data integrity:

  • Verify that all intended students have completed their assigned assessments and document any students who were absent or unable to complete testing

  • Document any technical issues for IT support to address systemic problems and improve future administrations

  • Note student concerns or difficulties reported during or after testing for consideration in score interpretation

  • Review data to confirm that score reports are generating properly and that results are available for authorized personnel

3.8.2 Score Interpretation Support

Effective use of assessment results requires ongoing support for educators in several key areas. Professional development should ensure that educators receive training in score interpretation and appropriate use of results for instructional planning. Contextual factors, including student background, testing conditions, and other relevant circumstances, should be considered when interpreting results. Multiple measures should be used together, with ROAR results serving as one component of comprehensive assessment data rather than standalone decision-making tools. Instructional planning should connect assessment results to specific instructional recommendations and interventions that address identified student needs.

3.9 Scheduling Recommendations

3.9.1 Optimal Timing

Several timing considerations can enhance the quality of assessment administration and student performance. Morning administration often yields better student engagement and attention, particularly for younger students. The day of the week matters, with mid-week administration generally preferable to Mondays and Fridays when students may be less focused. Academic calendar considerations suggest avoiding weeks with major events, holidays, or other disruptions that might affect student performance. Individual student factors, including attention spans, energy levels, and other needs, should be considered when planning assessment schedules.

3.9.2 Flexible Administration Options

The ROAR Comprehension Suite supports various administration approaches to accommodate different school contexts and student needs. Single session administration allows completion of all assessments in one 30-35 minute period, suitable for schools with longer class periods or dedicated testing times. Multiple session distribution across 2-3 shorter sessions over several days accommodates schools with shorter class periods or students who may need more frequent breaks. Individual pacing allows students to work at their own pace within time limits, supporting diverse learning needs and processing speeds. Make-up sessions provide opportunities for absent students to complete assessments, ensuring comprehensive data collection for all intended participants. These can be scheduled during independent work time, small group instruction windows, or designated assessment periods to minimize instructional disruption.

This flexible approach ensures that ROAR Comprehension Suite assessments can be integrated into diverse school schedules while maintaining standardized administration procedures that support valid and reliable results.