MOOC measurement problems reveal systemic evaluation challenges

Sadly, it’s not particularly surprising that it took a proclamation by researchers from prominent institutions (Harvard and MIT) to get the media’s attention to what should have been obvious all along. That they don’t have alternative metrics handy highlights the difficulties of assessment in the absence of high-quality data both inside and outside the system. Inside the system, designers of online courses are still figuring out how to assess knowledge and learning quickly and effectively. Outside the system, would-be analysts lack information on how students (graduates and drop-outs alike) make use of what they learned– or not. Measuring long-term retention and far transfer will continue to pose a problem for evaluating educational experiences as they become more modularized and unbundled, unless systems emerge for integrating outcome data across experiences and over time. In economic terms, it exemplifies the need to internalize the externalities to the system.

Not all uses of data are equal

Gil Press worries that “big data enthusiasts may encourage (probably unintentionally) a new misguided belief, that ‘putting data in front of the teacher’ is in and of itself a solution [to what ails education today].”

As an advocate for the better use of educational data and learning analytics to serve teachers, I worry about careless endorsements and applications of “big data” that overlook these concerns:

1. Available data are not always the most important data.
2. Data should motivate providing support, not merely accountability.
3. Teachers are neither scientists nor laypeople in their use of data. They rely on data constantly, but need representations that they can interpret and turn into action readily.

Assessment specialists have long noted the many uses of assessment data; all educational data should be weighed as carefully, even more so when implemented at a large scale which magnifies the influence of errors.

MOOCs plus big data

In The Coming Big Data Education Revolution, Doug Guthrie argues that “big data”, rather than MOOCs, represent the true revolution in education:

MOOCs are not a transformative innovation that will forever remake academia. That honor belongs to a more disruptive and far-reaching innovation – “big data.” A catchall phrase that refers to the vast numbers of data sets that are collected daily, big data promises to revolutionize online learning and, in doing so, higher education.

I agree that there are exciting new discoveries and innovations still yet to be made through the advent of big data in education, and I also agree that MOOCs’ current reliance on scaling up delivery of existing content isn’t particularly revolutionary. Yet I see the two movements as overlapping and complementary, rather than as competing forces.

While MOOCs may not (yet) have revolutionized instruction, they have revolutionized access for many learners. Part of their appeal for those interested in their growth is their potential for enabling large-scale analysis due to the high enrollments as well as the availability of online data. The opportunity to study such large numbers of students across such disparate contexts is rare in traditional academic settings, and it permits discoveries of learning trajectories and error patterns that might otherwise get missed as noise amidst smaller samples.

Another potential innovation which traditional MOOCs (xMOOCs) have not yet explored is new models of building cohorts and communities from amidst a large pool of learners, a goal at the heart of “connectivist MOOCs” (cMOOCs) that highlights peer-learning pedagogy. Combine xMOOCs and cMOOCs, and you can improve educational access even further by enabling courses to spring up whenever and wherever enough people, interest, and resources converge. Add in the analytical power of big data, and then you have the capacity to truly personalize learning, by providing both the experiences that best support students’ learning and the human interactions that will enrich those experiences.

Learner, Know Thyself

As “Big Data” loom larger and larger, the value of owning your own data likewise increases. Learners need to have access to all of their prior educational data, just as much as patients need access to all of their prior medical records, especially as they move between multiple providers and change over time. Instead of locking up valuable information in the hands of individual organizations with their own proprietary or idiosyncratic institutional habits, this lets the learner share their data for new educational providers to analyze.

Putting data back in the learners’ hands also empowers them to act as their own student-advocates, not just recognizing patterns in when they are learning more effectively (or less), but having the evidence to support their position. With accurate self-assessment and self-regulated learning becoming increasingly important goals in education these days, having students take literal ownership of their own learning and assessment data can help them make progress toward those goals.