Data Fueled Practice Exercises – SSAT
A Note from Test Innovators’ CEO, Edan Shahar
To date, students have answered over 20 million questions on the Test Innovators platform. Now, we are diving into this data to understand how students learn and to help them achieve their academic and test-taking goals.
After exploring how our students improve, we investigated further to learn how our various tools help students. We know our students improve by 36% after six practice tests, but we wanted to know which tools help them improve the most and how we can better adapt our platform to maximize its impact.
What Are Practice Exercises?
After a student completes a practice test, the Test Innovators system will generate specific, targeted recommendations for additional practice based on those test results. Practice exercises help students fill content gaps and improve their scores.
Practice Exercises Improve Scores
We initially analyzed the effect of practice exercises on overall student performance. We saw a five-percentile-point increase for students who complete at least one practice exercise compared to students who do not complete any practice exercises. We then examined the effect practice exercises have on student performance for each section, and we saw improvement across the board in every section; the math practice exercises for the Quantitative section have the biggest impact on improvement.
We discovered that students who complete math practice exercises not only increase their probability of correctly answering similar questions on the next test, but they also decrease the amount of time spent answering those questions.
Figure 1 depicts the percentage increase in the proportion of correct answers for section and difficulty matched questions from Test 1 to Test 2 when students completed or did not complete practice exercises between the tests. This shows a much greater increase in the proportion of correct answers after practice exercise completion. Note: The baseline proportion was higher for ‘Exercises Not Completed,’ which contributes to the difference in increased proportion.
Students who complete recommended practice exercises observe a much larger increase in the number of correct answers for questions of the same type and difficulty compared to students who do not complete practice exercises. It is important to note that students who do not complete exercises score higher on average on Test 1, which contributes to a smaller increase in correct answers on Test 2. However, the significant impact of practice exercises is evident, especially for students with lower percentile scores on Test 1.
Students Who Complete Practice Exercises
Students Who Do Not Complete Practice Exercises
Figure 2 displays the average time taken to answer an SSAT math question. Error bars represent the 95% confidence interval of the mean. We see a significant decrease in time taken to answer for students who completed practice exams (light blue) and a non-significant increase in time for students who did not (black).
Students who complete practice exercises don’t just improve their performance on similar questions, they also answer questions of the same type and difficulty on average 11 seconds faster on the second test than on the first test. Students who don’t complete practice exercises see little difference in time spent answering questions from Test 1 to Test 2. This suggests that students who spend more time answering questions and students who perform poorly on Test 1 may benefit the most from practice exercises.
Through our analyses, we learned that practice exercises can increase your performance and decrease the time spent answering questions. Additionally, we discovered that more practice is associated with higher achievement. But what does completing practice exercises do for long-term growth?
By looking at average student growth percentiles before and after a completed practice exercise, we can estimate increases in growth over time for students who complete practice exercises.
Figure 4 depicts Average Student Growth Percentiles before and after the completion of a practice sections. Error bars represent 95% CI.
On average, students who complete practice exercises see a three-point increase in Student Growth Percentile (SGP), increasing their math growth score from average to slightly above average.
More information about SGP can be found here.
A Note on Methodology from Test Innovators’ Data Scientist, Sean Coffinger
Analysis conducted with paired pre-practice exercises and post-practice exercises of similar difficulty for individual 9th-grade SSAT students in 2018. Performance increase analysis observed the proportion of students answering correctly prior to or after a practice exercise completion timestamp. We then looked at the percent increase in the proportion relative to the proportion of students answering a question correctly on the first attempt (Test 1). Pairs were matched by question type and difficulty. For question duration analysis, students who missed their first question, completed practice exercises, then answered a corresponding question correct on the following test showed a statistically significant decrease in question duration (paired t(540) = 4, p
To see data pertaining to ISEE students, please visit our Data Fueled Practice Exercises-ISEE blog post.
