7 ROAR-Syntax Assessment Design
7.1 ROAR-Syntax Assessment Design
ROAR-Syntax is a digital implementation and expansion of the Test for Reception of Grammar (TROG) (Bishop 1982), which has been used in clinical and research settings to assess syntactic knowledge. ROAR-Syntax maintains the original prompts, item format, and theoretical framework of the TROG. New items created by the ROAR team enhance measurement of syntactic structures at the upper end of the TROG’s original scope, shown in Table 7.2. The assessment was adapted for group administration with audio clips and updated illustrations, which maintain the core meaning and difficulty of the original items.
7.1.1 Assessment Format
ROAR-Syntax uses a four-alternative forced choice task to examine syntactic knowledge. An audio clip containing a sentence or short phrase is played through the student’s headphones, and four artist-drawn picture choices are simultaneously displayed on the screen. The student selects the picture that best matches the audio clip in meaning using one of three input methods: clicking with a mouse or touchpad, tapping on a touch screen, or pressing an arrow key on the keyboard. Prompts range from short phrases (e.g., “the red ball”) to full sentences (e.g., ‘The boy is jumping over the box’) (Figure 7.3). Pressing a replay button allows the student to hear the prompt as many times as they wish, and there is no time limit for answering any individual item. The assessment has an overall time limit of five minutes.
7.1.2 Practice and Instructions
Instructions are presented in English with the corresponding illustration choices for each sentence. The initial instructions describe how to provide answers: 1) “You can tap on a touchscreen, use the touchpad, or press the arrow keys for this activity.”, 2) “To use an arrow key to choose a picture, press the key that points the same direction as the arrow below the picture.”, 3) “You can press the replay button to hear things again.” The final instruction describes the main task: “This is a matching activity. Listen to what I say, then look at the pictures carefully. Choose the picture that goes with the words you hear.”
An arrow beneath each picture choice points in the same direction as the corresponding arrow key on the keyboard, helping students who use the keyboard understand which key corresponds to each choice. Practice questions with feedback reinforce the task structure before assessment begins, ensuring students understand what is expected before being assessed.
7.1.3 Example Item
Figure 7.1 shows an item from ROAR-Syntax, “The cow is pushing the lady.” Arrows under the picture correspond to arrow keys on the keyboard. The replay button is in the upper right corner of the screen.
The choices are (from left):
- Correct response: A cow pushing a woman
- Incorrect subject: A woman pushing another woman
- Incorrect action: A woman holding a bucket near a cow
- Syntactic contrast: A woman pushing a cow
7.1.4 Scoring
ROAR-Syntax items are automatically scored as correct or incorrect (dichotomous scoring) by comparing the participant’s response to the target answer that best represents the audio prompt. An item response theory (IRT) model converts the responses to an ability score (theta). The IRT model puts ability on an interval scale which enables scores to be compared over time and across grades.
7.1.5 Digital Adaptation
The original TROG was designed as an individually administered assessment consisting of 80 items organized into 20 blocks (Table 7.2), with 4 items per block covering progressively more difficult syntactic structures. For the TROG, a trained examiner administers the test to one student at a time and presents items in a fixed order until the student answers at least one item incorrectly in five consecutive blocks. While the TROG has strong psychometric properties and remains widely used in clinical settings, the individual administration requirement and test length limit its use for large-scale administration in school contexts.
For ROAR-Syntax, software was written to display illustrations, play and replay audio clips, and collect responses via mouse, keyboard, or touch screen. These features enable practical classroom administration while maintaining assessment validity. A graphic artist drew new images for all items, broadening the diversity of human characters. The new drawings were reviewed by researchers to ensure an accurate interpretation of the original illustrations. In particular, reviewers verified that the target answer was a clear example of the sentence and that distractors corresponded to the same linguistic and syntactic errors as the original version. Audio clips were created using AI voice generation (ElevenLabs) with standardized speech rate and prosody and these clips were reviewed for accuracy and clarity.
7.1.6 Computer Adaptive Testing
For all studies, items were presented in random order (all items were equally likely to be presented to both low and high performers) to facilitate item-response analysis and the creation of an IRT model. After item analysis is complete, the IRT model will be used to implement computer adaptive testing.
7.1.7 Item Modifications and Exclusions
The first two items of TROG block A, a vocabulary check for nouns (e.g., ‘shoe’, ‘bird’), were repurposed as practice items for ROAR-Syntax. An optional extended vocabulary check in the TROG was not included in ROAR-Syntax. The original TROG prompts were reviewed for language that could be problematic in a school setting. Three items were changed: “The boy chasing the horse is fat,” was changed to “The boy chasing the horse is tall,”, “The knife is longer than the pencil,” was changed to “The fork is longer than the pencil,”, and “The knife is on the shoe,” was changed to “The fork is on the shoe.” The ROAR team chose not to include the four items from block I, assessing ‘masculine/feminine personal pronouns’, because answering correctly would require reliance on gender stereotypes of clothing and hairstyles to select pictures corresponding to ‘he,’ ‘she,’ ‘him,’ and ‘her.’ This exclusion reflects a commitment to reducing construct-irrelevant variance related to gender stereotypes. From the original 80 TROG prompts, ROAR-Syntax used 2 items for practice, excluded 6 items, lightly modified 3 prompts, and included 69 of the original prompts unchanged.
7.2 Development of the Construct Map
The TROG was originally created by Bishop (1979) as a research tool to investigate whether children with specific language disorders whose expressive speech was syntactically atypical had similar problems in understanding syntactic contrasts. It was subsequently revised and standardized to provide a test suitable for clinical use in children from 4-12 years of age. Bishop took particular care to include structures known to be problematic for children with language disorders. The TROG items are organized into 20 blocks with four items each (Table 7.2). Every block examines understanding of a specific syntactic structure by presenting contrasting interpretations of a sentence prompt. For example, for the item “The cow is pushing the lady,” one choice shows a cow pushing a woman while another shows a woman pushing a cow (Figure 7.1). To facilitate administration and scoring, Bishop ordered the blocks by difficulty (taking into consideration distractor complexity) and labelled them A through T (Bishop 1982). ROAR-Syntax adds new blocks enhance the upper end of the test with additional structures that are prominent in academic text (Bailey 2007).
7.2.1 Construct Map of Syntactic Knowledge
The ROAR-Syntax construct map (Table 7.1) was created by analyzing the TROG scope and sequence and enhancing it with additional syntactic structures that are frequently used in academic texts.
| Waypoint | Description | Evidence of Understanding |
| 0 - Not yet evident. | The student does not yet demonstrate comprehension of syntactic structures in English. | The student chooses illustrations that contain objects or actions that are not in the prompt. |
| 1 - Developing | The student understands some syntactic structures that are common in spoken English and is beginning to understand some structures found mainly in academic texts. | The student chooses the correct match between illustrations that depict the same content words but differ in one or two syntactic structures.
The student can match a word, phrase, or sentence containing 3 or fewer elements in subject-verb-object word order to an illustration that contains all objects and/or actions in the prompt.
The student is developing understanding of the following structures: active voice, pronoun relationships, plural or comparative inflections, coordinating conjunctions,passive voice,spatial relationships, postmodified structures, complex negation patterns, subordinating conjunctions, gerund phrases, and relative phrases. |
| 2 - Advancing | The student understands most syntactic structures that are common in spoken English, and is continuing to develop understanding of structures found mainly in academic texts. | The student chooses the correct match between illustrations that differ by two syntactic structures. The student is developing understanding of the following structures: advanced prepositions, embedded phrases and clauses, and participle phrases. |
| 3 - Proficient | The student understands most syntactic structures in the assessment, including those found mainly in academic texts. | The student correctly interprets prompts that contain embedded phrases and clauses, advanced prepositions, and participle phrases. |
7.2.1.1 Waypoints
Not yet evident
Students at this level do not show understanding of syntactic structures that are common in spoken English. Some potential reasons for scoring at this level include very low levels of English vocabulary, disengaged / random responding, or equipment issues (e.g. broken headphones).
Developing
Students at this level correctly interpret some syntactic structures that are common in spoken English. As their skill increases, they are developing understanding of the following structures that are common in spoken English: active voice, pronoun relationships, plural or comparative inflections, and coordinating conjunctions. They are also developing understanding of structures found mainly in academic texts: passive voice, spatial relationships, postmodified structures, complex negation patterns, subordinating conjunctions, gerund phrases, and relative phrases.
Advancing
Students at this level correctly interpret most of the syntactic structures that are common in spoken English. Their skill is advancing to understand more complex structures found mainly in academic texts: embedded phrases and clauses, advanced prepositions, and participle phrases.
Proficient
Students at this level correctly interpret most syntactic structures in the assessment, including ones that are mainly found in academic texts such as embedded phrases and clauses, advanced prepositions, and participle phrases.
7.2.2 Syntactic Structures
The construct map is operationalized by blocks of items that target specific syntactic structures (Table 7.2). Blocks A-T contain the syntactic structures specified in the TROG manual (Bishop 1981) while blocks AP - SC contain additional syntactic structures added by ROAR researchers.
| Syntactic Structure | Block | Example |
| Single-element (Noun, Verb, Adjective) |
A-C | Choose the picture that shows eating. |
| Two-element combination | D | The boy is running. |
| Negation | E | The boy is not running. |
| Three-element combination | F | The boy is jumping over the box. |
| Singular/plural personal pronoun | G | They are sitting on the table. |
| Reversible active voice | H | The girl is pushing the horse. |
| Masculine/feminine personal pronoun (not used) |
I | She is sitting on the chair. |
| Singular/plural noun inflection | J | The cats look at the ball. |
| Comparative/absolute | K | The fork is longer than the pencil. |
| Reversible passive | L | The girl is chased by the horse. |
| In and on | M | The cup is in the box. |
| Postmodified subject | N | The boy chasing the horse is tall. |
| X but not Y | O | The box but not the chair is red. |
| Above and below | P | The pencil is above the flower. |
| Not only X but also Y | Q | Not only the bird but also the flower is blue. |
| Relative clause | R | The pencil is on the book that is yellow. |
| Neither X nor Y | S | Neither the dog nor the ball is brown. |
| Embedded sentences | T | The book the pencil is on is red. |
| Advanced prepositions | AP | The fish swim beneath a whale and a sea turtle. |
| Coordinating conjunction | CC | The kids cleaned the room, but forgot to put away the train. |
| Compound Predicate | CP | She tripped on a rock and dropped her books. |
| Gerund phrase | GP | Bumping the table with my foot caused a book to fall. |
| Participle Phrase | PP | The girl wearing a backpack was shown a flower by her friend. |
| Subordinating Conjunction | SC | The teacher will give the students cake if they stand in a line. |
The easiest blocks of the TROG probe basic listening comprehension in English. Blocks A-C are a vocabulary check for some of the nouns, verbs, and adjectives used later in the test. Blocks D-F target understanding of simple sentences with basic word order in English (subject-verb, subject-verb-object) and negation of verbs (“not running”).
The next group of blocks require attention to syntactic markers rather than vocabulary matching alone. Blocks G-L assess understanding of pronouns, active and passive voice, plural nouns, and comparative adjectives. In particular, blocks H and L probe understanding of word order using sentences where either the subject or the object could logically perform the action.
Success on the later blocks demonstrates command of structures typically encountered in academic texts. Blocks M-T assess understanding of prepositions (“in”, “on”, “above”, “below”), complex negation patterns (“but not”, “not only…but also”, “neither…nor”), embedded phrases, and relative and embedded clauses.
7.2.3 Enhancing the TROG
The ROAR team used developmental progressions of oral and written language skills to identify syntactic structures that are not assessed by the TROG. They created new items and grouped them into the following blocks:
- AP: Advanced preposition (3 items)
- CC: Coordinating conjunction (3 items)
- CP: Compound predicate (2 items)
- GP: Gerund phrase (1 item)
- PP: Participle phrase (2 items)
- SC: Subordinating conjunction (7 items)
Additional items covering these structures will be introduced in the 2025-26 school year.
During analysis, researchers reviewed the new items to verify that they were assigned to the correct blocks. Five items (not included in the counts above) matched existing TROG blocks:
- R: Relative phrase (3 items)
- S: Neither-nor (1 items)
- T: Embedded sentence (1 item)
7.2.4 Example researcher-created item
Figure 7.2 shows an example of a researcher-created item that contains two independent clauses and a coordinating conjunction. “Although it is hot outside, I am wearing a jacket with a hood.” It assesses both the understanding of “although” and the distinction between “with” and ”without”. The choices are (from left):
- Incorrect first clause: cold scene, jacket with a hood
- Two incorrect clauses: cold scene, jacket without a hood
- Incorrect second clause: hot scene, jacket without a hood
- Correct response: hot scene, jacket with a hood
7.3 Task Format Rationale
Consistent with the TROG, in each ROAR-Syntax item the student listens to a sentence and chooses the answer from 4 illustrations. This format supports both psychometric quality and practical implementation.
First, the format mirrors authentic reading comprehension demands. When students encounter complex language in text, they must correctly interpret word order and syntactic cues from function words and suffixes to accurately determine meaning. In ROAR-Syntax, students must understand the meaning of a sentence well enough to distinguish it from plausible alternatives, providing greater ecological validity than decontextualized manipulation tasks.
Second, the systematic distractor design reveals not just whether students understand a syntactic structure, but how they process grammar. Distractors represent plausible but incorrect interpretations of the stimulus sentence. This ensures that selecting the correct answer requires genuine syntactic comprehension, not elimination of absurd choices or guessing strategies. Distractors are designed to correspond to lack of syntactic knowledge, inattention to word order, or focus on lexical content rather than syntax. Response options that are systematically varied provide better discrimination among ability levels and more valid ability estimates (Briggs et al. 2006).
Third, from a practical standpoint, the multiple-choice format enables automated scoring and efficient group administration without requiring specialized training. Teachers administer ROAR-Syntax through the ROAR platform, which automatically scores responses and provides results. The format does not require specialized test-taking skills and reduces construct-irrelevant variance that might interfere with accurate measurement.
7.3.1 Distractor Design
The original TROG items use three distinct distractor design strategies: lexical-only, lexical-syntactic, and multi-syntactic (Table 7.3). Items created by ROAR researchers use the muli-syntactic distractor strategy.
| Distractor design | Blocks | Description |
| lexical-only | A-D, F | Choices consist of the correct answer and three lexical distractors – illustrations where one word (a noun, adjective, or verb) differs in meaning from the prompt. (Figure 7.3) |
| lexical-syntactic | E, G-L | Choices consist of the correct answer, two lexical distractors, and a syntactic distractor – an illustration that depicts the same nouns and verbs but differs according to syntactic relationships determined by word order, prepositions, conjunctions, or function words. (Figure 7.4) |
| multi-syntactic | M-T AP, CC, CP, GP, PP, SC |
Choices consist of the correct answer and three syntactic distractors. All choices depict the same objects, which differ by color or spatial relationship. Two syntactic structures are varied independently: either both are correct, only one is correct, or neither is correct. (Figure 7.5 and Figure 7.6) |
lexical-only
Choices consist of the correct answer and illustrations where one noun, adjective, or verb differs from the prompt. These items check for semantic understanding of very basic words and simple phrases; they do not test syntactic knowledge. For instance, in “The boy is jumping over a box” (Figure 7.3), the choices are (clockwise from top):
- correct response (a boy jumping over a box)
- lexical distractor (a boy holding a box)
- lexical distractor (a boy jumping over a flower)
- lexical distractor (a horse jumping over a box)
lexical-syntactic
Choices consist of the correct answer, two lexical distractors (where a content word differs from the prompt), and a syntactic distractor (where the illustration differs from the prompt by a single syntactic contrast).
For example, in “The elephant is pushed by the boy,” (Figure 7.4), the choices are (clockwise from top):
- correct response (a boy is pushing an elephant)
- syntactic distractor (“is pushed by” is misinterpreted as the elephant pushing the boy)
- lexical distractor (the boy is showing an apple to the elephant)
- lexical distractor (the boy is pushing a truck)
multi-syntactic
All choices depict the same objects (there are no lexical distractors). The objects in the illustration differ by color or spatial relationship. Two syntactic features are varied independently: either both are correct, only one is correct, or neither is correct. This design ensures that students cannot answer correctly through simple word-matching; they must understand both syntactic relationships expressed in the sentence.
For example, to select the correct answer for “The cow chasing the cat is brown,” (Figure 7.5) it is necessary to identify which animal “is brown” and also to determine which animal is “chasing”. The choices are:
- correct response: (a brown cow chasing a white cat)
- syntactic distractor: (the wrong animal is brown)
- syntactic distractor: (the wrong animal is chasing)
- syntactic distractor: (both errors).
The multi-syntactic strategy is also used for some items involving negation. For example, to select the correct answer for “The box but not the chair is red,” (Figure 7.6) it is necessary to understand the meaning of “but not” and also to identify which object is referred to by “is red”. Researchers used a website that simulates color-blindness to verify that color-blind individuals would be able to distinguish between the choices.
The choices show:
- correct response (a yellow chair and a red box)
- syntactic distractor (the wrong object is red)
- syntactic distractor (both objects are red)
- syntactic distractor (neither object is red)
7.4 Item Development and Quality Assurance
7.4.1 Item Review
Before items were administered to students, a panel of researchers and educators reviewed all prompts and illustrations according to the following criteria: construct alignment (does the item measure the intended syntactic construct rather than vocabulary, world knowledge, or visual interpretation?), linguistic accuracy (are syntactic transformations correct and distractors genuine alternatives?), and cultural sensitivity and inclusivity (are items free from bias and do illustrations reflect diverse representations?).
Items were revised based on panel feedback. Common revisions included vocabulary changes, illustration adjustments, clarification of syntactic contrasts, and distractor refinement. This iterative review process ensured that all items met rigorous standards.
7.4.2 Item Development Timeline
In the 2023-24 school year, version A of ROAR-Syntax included nine researcher-created items. In the 2024-25 school year, version C introduced fourteen more researcher-created items. We plan to add thirty-two additional items in the 2026-27 school year.