Predictive Validity and Cutoff Scores of Acadience Reading Assessments for Third Grade RISE Reading Assessments

The Utah State Board of Education (USBE) partnered with the Utah Education Policy Center (UEPC) to evaluate the predictive validity of the Acadience Reading assessment’s four subscores (Oral Reading Fluency, Accuracy, Retelling, and MAZE), measured in first through third grade, in reference to two subscores on Utah’s third grade RISE Reading assessment: Informational Text and Literature.

In addition, the UEPC evaluated the Reading On Grade Level (ROGL) cutoff for the Acadience Reading comprehensive scores in first through third grades with regard to the benchmark for proficiency on the RISE English Language Arts Assessment in third grade. Across Acadience subscores in first, second, and third grade, the average percentage of variance in RISE subscores explained was 23% (ranging from 10% to 35%). However, the Acadience subscores were not equal in their predictive ability.

Fluency was consistently the strongest single predictor of both RISE reading subscores (Informational Text and Literature) in all three grades. Predictive power was marginally improved if MAZE was used together with Fluency to predict RISE scores (percent of variance explained improved between 1.87 and 2.24 percentage points).

Predictive validity showed few differences across gender, economic disadvantage, and disability status. However, five groups had predicted RISE scores that were more than 0.1 standard deviations higher than their actual RISE scores (i.e., they were consistently overestimated): English Language Learners, students who are Black or African American, Hispanic students, Native American students, and students who are Pacific Islander or Native Hawaiian. This pattern is concerning because it suggests that forecasts made based on Acadience subscores for these students will tend to overestimate their actual RISE Reading scores, putting them at greater risk than their reference groups of being classified as proficient based on an Acadience test and then later not passing the RISE test (i.e., of being “False Passes” in the terms introduced in the next section).

Overall, the current cut scores in Appendix G of the Utah Accountability Technical Manual for first through third grade perform relatively well at predicting which students will be categorized as Reading on Grade Level at third grade from the RISE ELA assessment, with all three showing Accuracy above 78%. However, consideration should be given not only to overall accuracy but also to the relative costs of False Passes and False Fails.

View report

 

Search