A/B testing is used throughout the tech world to build more engaging and valuable products. By randomizing users into either the current baseline experience (control) or a new experience (treatment), companies can measure the causal effect of the new experience on the outcomes they care most about.
Coursera’s platform enables instructors and researchers to bring that same technology and methodology to pedagogy. We do this by randomizing enrollees into different versions of the same course …
… and then observing the effect of different teaching inputs (such as videos) on learning outputs (such as course completion).
Our approach has proven powerful in a range of contexts:
- It has informed content presentation formats. For example, instructors have used our infrastructure to test videos vs. slideshows vs. reading for optimal delivery of a given topic.
- It has informed how we balance formative with summative assessments, in particular because formative assessments have been found to improve learning outcomes.
- It has informed how we motivate and reward learners, and reminded us to pay careful attention to psychological factors when designing our platform. For example, instructors have found that having learners state a goal upfront improves course progression, especially among historically disadvantaged groups.
Coursera instructors have more than a dozen experiments live on the platform today, and they — and we — are closely monitoring the results, and adapting the platform and content based on what we learn. Here are two examples from inquiries our partners are independently conducting on inclusion:
1. Chris Brooks, from the University of Michigan, wants to know: Do subtle gender cues impact female learners in STEM content? He randomized the appearance of male versus female workers in the background of some videos, and found that this had an impact on engagement with the material presented, but not on downstream retention or course completion.
2. Rene Kizilcec, from Stanford, is interested in the question: Can we mitigate a social identity threat (when people feel too estranged from the social context of their learning environment to relax and learn)? He randomized whether learners received focused interventions, for example whether they were asked their goals at the beginning of the course and then reminded throughout. His results suggest that mitigating social identity threat can reduce the global achievement gap.
By empowering our partner community to experiment, we hope to accelerate the pace of pedagogical innovation on the platform, bringing us ever closer to achieving our mission of transforming lives through the world’s best (and most thoroughly A/B tested!) education.
. . .
Interested in applying data science to education? Coursera is hiring!