“Big Breakthroughs Happen When What Is Suddenly Possible Meets What Is Desperately Necessary” – Thomas L. Friedman

Toronto, Ontario. I am attending the annual conference of the Association for the Advancement of Artificial Intelligence (AAAI). This year roughly 1,000 attendees have converged on Toronto to hear about and see demonstrations of the latest advances in building “intelligent” computing systems.

This morning I attended Andrew Ng’s talk on “The Online Revolution: Education at Scale.” For those of you who may not have been following the shake-up that’s happening in higher education, Andrew is at the heart of the revolution. In the fall of 2011, he taught his Stanford University course on Machine Learning to 400 students in class, simultaneously with 100,000 students online. That course (and the Artificial Intelligence course taught by Sebastian Thrun and Peter Norvig) ignited a firestorm, generating massive international media attention on MOOCs – massive open online courses. The result was that Andrew (with colleague Daphne Koller) founded Coursera and Thrun started Udacity, both companies having the goal of bringing superior educational courses to the world via online technology.

Here is a brief summary of the key points in Andrew’s talk. Warning: this is a much longer post than usual. There is lots to talk about!

Motivation

Andrew argues that world-wide there is a shortage of opportunities for getting access to high-quality higher education. There are many reasons for this, including financial obstacles and limited enrolments. By offering courses online for free, both barriers get razed. His online Machine Learning course reached an audience 250 times larger than his traditional in-class audience. He did not say how many successfully completed the course. I understand from other sources that it was around 5,000. Still, getting that many people to pass an advanced highly-technical course is stunning.

Of course, there is a difference: a certificate for passing an online course is not the same as academic credit towards a Stanford degree. However, in terms of learning outcomes, the point is mute.

Of interest is that he received appreciative feedback from people around the world, such as from a 39-year-old single mother in India who had never dared to dream of taking a Stanford University course. When I went to talk with Andrew after his lecture today, I had to wait in line. Most of the 20 people ahead of me were international students who had enrolled in his online course. They wanted to thank him in person for their excellent learning experience.

In less than a year, Coursera has had 800,000 registrants from 190 counties for a total of 2 million course enrollments in the 111 online courses offered (science, humanities, business, etc.). No word on how many people passed the courses. Regardless, by any standard these are impressive numbers.

Andrew noted that many people did not finish a course because “their life got busy”; they could not sustain the three-month intensive experience. Coursera is considering offering courses at “half speed,” to spread the workload over a longer period.

Secrets of Success (1): Video-based Instruction

I have looked at courses offered by Udacity (specially filmed) and Coursera (professor lectures), and been more impressed with the Udacity production values. However, Andrew argued that the lecture approach is critical to success. Instructors can create lectures in their home or office; all they need is a quiet space with a computer and web cam. This avoids the expensive production infrastructure that, presumably, Udacity invests in. The argument is that this is the easiest way to quickly scale up the number of courses available since, essentially, it enables anyone to prepare an online course.

Coursera doesn’t offer the traditional hour-long lectures. Instead the material is broken into 10-minute “bite-sized” chunks, allowing students to more easily absorb the material.

Students are presented with optional pre-requisite material (for those who need to refresh their background skills) and optional advanced material (for the keeners). This allows Coursera to say they have moved away from the one-size-fits-all model offered by most online courses.

Secrets of Success (2): Assessment

Andrew argues that the online world allows for novel assessment opportunities. He did not claim this, but I inferred that he believed they were superior to traditional assessment models. He raised several important points:

• videos can have test questions interspersed, allowing the student to pause (for as long as they want) and test whether they understand the material;
• Coursera uses extensive software-based tools to validate student answers;
• students can attempt a problem as many times as needed until they get it right;
• additional test questions can be automatically generated;
• the online system can be adaptive so that when a student makes a mistake, they are pointed to the relevant instructional material; and
• use of a peer grading system.

The last point is critical to their model. The education literature, as well as Coursera’s own research, says that peer grading can be highly correlated to grading done by the instructor. By combining peer grading with crowd sourcing, Coursera can scale up to accommodating (hundreds of) thousands of students.

Students must attend a grading boot camp. They demonstrate their proficiency by assigning marks to assignments that have already been graded by the instructor. If their result closely matches that of the instructor, then they are allowed to grade other students’ work. For each course assignment, every student is expected to grade five and, in return, gets feedback from five. Coursera has data to say this produces high-quality results.

Secrets of Success (3): Community

The global nature of the course audience means that students have 24x7 access to course assistance. A support community quickly builds, with students helping each other. For the Machine Learning course, the median time for a student question to be answered by someone was 22 minutes – much better than anything I could ever do in any course that I have taught.

Perhaps surprisingly, some of the online study groups translated into face-to-face study groups, including two in China, three in India, and one in London. A study group has recently been set up in Palo Alto and 1,000 people have signed up!

Secrets of Success (4): Statistics and Analytics

I have read several papers in the education literature that involve experiments with human subjects. They usually involve small samples (as low as five, and rarely more than 100). As a scientist who is used to working with large data sets, I find these papers unsatisfying. Contrast that with what Coursera is doing. Because of the large enrolments, they have the opportunity to do some amazing analysis. For example, to test a hypothesis they will use data gathered from 20,000 students, with a further 20,000 as a control group. They collect anonymized data on every student mouse click, key stroke, time spent reading a web page, number of times questions were answered incorrectly, how often a video is watched, and so on. They mine this data to better understand the pedagogy of their courses – what works and what doesn’t work. Andrew claimed that what Amazon did for e-commerce, Coursera will do for e-education.

He gave one interesting example of how data mining can work. On an assignment, the system identified that 2,000 students submitted the same wrong answer. After manually looking at the incorrect solutions, it became clear that there was a trivial misunderstanding. The online system was then modified to detect when the mistake is made and then point the student to a web page that hints at what they are doing wrong. Coursera is working on automating this process.

Conclusions

Andrew raised an interesting characterization of how the online world differs from the in-class experience. In a traditional course, time is held constant (you need to complete something by a specific date); the amount you learn is variable. In a Coursera course, the amount you learn is a constant (you can try as many times as you like until you get it right); the time you spend on the course is variable. He believes (and may have data to support his claim) that the amount learned per student in his online course was higher than for his in-class course.

Whether you agree or disagree with the move towards online education is irrelevant; it;s here and it's not going away. How often does university teaching attract extensive international media attention? These massive open online courses may be the disruptive technology that will shake the traditional educational system to its very foundation. It is too early to know the full implications of what is happening, but it is obvious that we are only feeling the tremors of change right now; the full seismic impact is yet to come.

Let me conclude with a quote from the New York Times: “This is the tsunami,” said Richard A. DeMillo, the director of the Center for 21st Century Universities at Georgia Tech. “It’s all so new that everyone’s feeling their way around, but the potential upside for this experiment is so big that it’s hard for me to imagine any large research university that wouldn’t want to be involved.”