Woot Math Adaptive Technology Efficacy Study
Supported by:
The National Science Foundation
Download the full report:
Supplemental documents:
Effective Personalized Learning
Woot Math uses advanced technology to adapt to each student’s needs and to deliver genuine, personalized learning. While many products claim to deliver benefits through personalized learning, there has been very little empirical evidence that any in fact do (Shute & Zapata-Rivera 2012, p. 22; Izumi, Fathers, & Clemens, 2013, p. 7).
In the Spring of 2014, with funding from the National Science Foundation (NSF), we conducted a multi-site randomized controlled trial that studied several forms of software-based adaptivity and demonstrated that Woot Math’s adaptivity contributed significant effects in learning and retention. This web page provides a summary of the study and its findings; full reports detailing the study are available as pdfs from the sidebar on the left.
In the trial, the only difference being tested was that the experimental adaptive capabilities were enabled for the test group but disabled for the control group. The study results show statistically significant evidence that the adaptive version improved learning and retention, with moderate to large effect sizes (g = 0.23 to g = 1.50, p < .05) in a sample of 350 students in grades 3-6. (The study was conducted during the last six weeks of the school year, so the participating students had nearly completed the grade for which their results are reported.)
Research Goals
The two questions that our study was intended to answer were the following:
- Is there evidence that an adaptive version delivers better learning and retention?
- Is there evidence that the adaptive version either positively impacts student sentiment or leaves it unchanged?
We also examined teacher and student sentiment about their experience with Woot Math as a whole (independent of their assignment to test or control groups, i.e., independently of whether their experience included the experimental adaptive features or not).
Methods
The study implemented a parallel group randomized controlled trial at four demographically diverse sites, and was designed to meet the What Works Clearinghouse standards for scientific evidence for what works in education. Student participants were randomized into the test and control groups in equal numbers, and the trial was fully blinded – students, teachers, and investigators were not aware as to which group each student participant had been assigned. Both groups (test and control) completed a treatment intervention using Woot Math, with the only variation being the use of Woot Math's adaptive capabilities. Students in the test group were given adaptive instructional sequences and the control group non-adaptive sequences (i.e., as linearized by content experts). By isolating the effects of Woot Math's adaptive features, we were able to answer the research questions posed above.
At the end of the trial, participating students were given a summative assessment on the material covered (ordering and equivalence of fractions). The summative assessment was designed by the research team and is described in detail in the White Paper. The majority of assessment tasks (64%) required students to construct a response – e.g., drag a fraction to the correct location on a number line, model a fraction using a bar model, etc. Considering the grade level of participating students, the reasoning required to successfully respond to these tasks was not trivial.
To assess student and teacher sentiment, surveys were conducted after the summative assessment and the participating teachers were also interviewed.
Findings
The participating students who were given the adaptive treatment scored significantly higher on the summative assessment than did the participating students in the control group. This primary outcome finding is illustrated in the figure below.
Survey Results
In addition to statistically improved outcomes, we wanted to show that using Woot Math can improve students' attitudes toward math. The data from the student and teacher surveys strongly supports this conclusion. The tables below summarize a selection of survey responses.
Student Survey Responses
Students responded very favorably to Woot Math, both in the test and control groups, indicating a high level of engagement and sentiment toward the treatment. Additionally, students across the range of grades in the study (3rd to 8th Grade) considered the application to be helpful and age appropriate.| Student Survey Question | Responses | |
| How helpful did you find Woot Math? | 95% "Helpful" | 67% "Very helpful" |
| Did you learn new things about fractions? | 90% "Yes" | 50% "Yes, a lot" |
| Do you feel more confident about math after using Woot Math? | 90% "Yes" | 52% "Yes, much more" |
| Would you say the problems were: | 79% "Right difficulty" | |
| Would you recommend Woot Math to another student? | 96% "Yes" | 65% "Yes, for sure" |
Teacher Survey Responses
Teachers were very positive in the survey, finding Woot Math to be instructive and helpful, increasing students' confidence in math by providing successful independent learning.| Teacher Survey Question | Responses | |
| How instructive did you find Woot Math to be for your students? | 100% "Instructive" | 70% "Very instructive" |
| Do you think it may have improved their attitudes or self-confidence about math? | 94% "Yes" | |
| Do you think it increased their confidence for working and learning independently? | 94% "Yes" | |
| For what portion of your students was it helpful? | 94% "More than 70%" | |
| For your students, would you say the material was: | 81% "Just right" | |
Teacher Interviews
As part of the study, final interviews were conducted with each of the participating teachers. Transcripts of all seventeen interviews are available for download below.
A number of common themes emerge from the full set of interviews. The quotes below are taken directly from these transcripts and highlight those themes:
Adaptive and Engaging
I liked that Woot Math adapted to their needs. Neither my low nor my high students complained. That is remarkable. It is really hard as a teacher to differentiate across the class.” —4th Grade Teacher
I really liked that it was adaptive to each student. I have such a wide variety of students, but they could all do it - not only could they do it, but they could all be successful.” —4th Grade Teacher
As a classroom teacher I really like that they were engaged. I was better able to focus on small groups, and it gave me the time to do more effective one-on-one help.” —5th Grade Teacher
I like how engaged they are - I knew they weren’t going off topic or losing focus. It is a great tool to have - totally independent learning, with high focus that everyone can do.” —4th Grade Teacher
Novel, Hands-on Approach
Fractions are really hard for students to grasp and for teachers to teach. I think Woot Math is not only transformative for the students; it is also transformative for the teachers.” —5th Grade Teacher
I liked that Woot Math exposed my students to fractions in new ways... I liked the different formats that it presented problems in. It helped teach them that they can break everything down.” —4th Grade Teacher
Fractions are always a struggle - and just the fact that they are so fully engaged while hands-on working with fractions and solving fraction problems was incredible.” —6th Grade Teacher
Classroom Implementation
Woot Math really helped change what we are doing in math class. In my class the girls were silent and the boys dominated the conversation. By dividing the class by gender, half on Woot Math and half with me, the girls’ conversation became so much richer than before. They weren’t talking about their day, they were talking about math! Our principal observed our class one day, and he said it was the best math lesson he had ever seen taught at our school. This would not have been possible without something like Woot Math.” —5th Grade Teacher
Woot Math is a perfect 4th grade math center... It is really easy to use Woot Math with no teacher input, and I could focus on the other groups.” —4th Grade Teacher
I love the captured screens from the teacher dashboard. I also like that Woot Math sees their mistakes and tailors to their needs.” —4th Grade Teacher
Conclusion
As detailed above, the results of our study provide clear evidence that Woot Math's adaptive techniques improve student outcomes. Our results establish a causal relationship – that the addition of these innovative software formative assessment techniques resulted in improved learning outcomes. These results were also supported by a wide range of anecdotal evidence from student and teacher surveys and interviews indicating the overall effectiveness of the product.
