The MOOC Certificate Paradox

As I’m starting to review some of the courses that I’ve taken, one of my goals is to let prospective employers and admissions offices what they should think when they see a particular MOOC certificate on an applicant’s resume or application. That assessment generally consists of three parts: identity verification, assessment rigor, and content coverage.

Generally, few current MOOC providers feature sufficient verification to make the certificate all that impactful. Coursera’s identity verification, for example, provides no evidence that the student completed the work instead of receiving it from a friend or a database online. Solutions exist to both the problems of identity verification and plagiarism, but thus far, I have not seen many (well, any) MOOC platforms use them.

In the short term, that’s not a major problem. These certificates aren’t yet powerful enough to bother cheating on them. If I’m not going to get significant credit on a job or school application for completing a course, why bother cheating to complete it? This sets up what I call the MOOC Certificate Paradox: the value of a MOOC’s certificate is inversely proportional to the value of the MOOC’s certificate.

How can that be possible? So long as the certificate has no value, we can infer that students only complete it for personal improvement and learning. So long as we can believe they only complete the MOOC for personal improvement and learning, we can attach value to their completion of the MOOC: we know they were not simply earning the credit to improve their resume or application because earning the credit wouldn’t improve their resume or application. Thus, the certificate has value. But if the certificate has value, there emerges an incentive to earn it specifically to obtain that value. We no longer know that they were not simply earning the credit to improve their resume or application because now earning the credit does improve their resume or application. Thus, the value of the certificate diminishes when its value increases. The value of a MOOC’s certificate is contingent on it not having inherent value.

However, this stands in opposition to what I believe to be the function of Verified Certificates and other similar certification programs: to make the course mean something.  Verified Certificates attempt to attach value to MOOC completion, but in the absence of firmer measures to verify students’ identities and prevent plagiarism, that value cannot arise: the moment it arises, it diminishes as an incentive to cheat (as well as an absence of any reliable way to identify cheaters) is introduced.

Of course, there is a way to break out of this paradox: there must be more rigorous methods to verify identities. Quizzes and exams in MOOCs need to be legitimately proctored. All submissions need to be automatically checked for plagiarism. The presence of these measures will provide inherent value to MOOCs rather than value that is dependent on the student’s motives.

Starting Five New Online Courses

Five new online courses start for me today:

Given that I’ll be posting reviews for all courses I take at the conclusion of the course, I won’t be posting first impressions posts anymore. However, reviews for my first four courses should be up in the next couple weeks.

Are open online education and quality online education mutually exclusive?

In the past, I’ve touched on a distinction I see in the landscape of higher education. It is this distinction that leads me to say that programs like Coursera and edX and programs like Udacity and the Georgia Tech OMS are not competitors, but rather represent two largely different goals of education: openness and quality.

Of course, I hate using the word ‘quality’ because it implies that open education cannot be high-quality, which is not what I mean to suggest. Rather, what I mean to suggest is that openness and quality often get in the way of one another. Developing open courses for a platform like Coursera almost inherently dictates that costs must be extremely limited. Offering a course through Coursera does not bring in a tremendous amount of money; even the Verified Signature track, I would speculate, barely pays for the human effort required to grade assignments and verify identities. Developing open courses can be an act of either marketing or altruism, but in either case, there is a natural impetus to keep costs low. The outcome, of course, is nonetheless fantastic: the world’s knowledge presented by the world’s experts on that knowledge in a venue that everyone can access. Even if the cost pressure demands that this information can only be presented in the traditional lecture model, the outcome is nonetheless incredibly desirable.

That openness is largely driven by the internet’s ability to deliver content to massive audiences for low costs. However, that’s not the only thing that the internet can do in service of education. The internet also has features and frameworks that can create educational experiences that go beyond what we can do in traditional classrooms. Many traditional college classes are delivered in the same lecture model as the aforementioned Coursera courses, but pedagogically we know that this model is largely ineffective. It is not chosen because it is effective, however; it is chosen because professors’ time is valuable, professors are very often experts in the subject matter rather than in teaching itself, and the lecture model is arguably the easiest way to present material. There are exceptions, of course, but I don’t think I’m being controversial in suggesting these ideas as generally true.

What the internet gives us, however, is a mechanism by which content can be produced once to be consumed by millions. This is part of the reason the openness initiatives work: professors can film the course once and make it available to the masses rather than having to reteach it semester to semester. But while in some places that is an impetus for openness, we may also use that as an impetus for quality. Let’s invent some numbers to make it clearer. Let’s imagine that a class of 50 students are each paying $100 to take a class; this means that the class must cost no more than $5,000 to deliver each semester. However, if the class could be developed once and re-used ten semesters in a row, that means that the same class now can cost up to $50,000 to develop, allowing for much more investment into the quality of the class.

This, of course, is a gross simplification, but it is intended to portray an elegant truth: when we use the internet to deliver content to a much larger population with the same amount of work, we can either pass on the savings to the students (the openness route), or we can reinvest the money into the development of the courses themselves (the quality route). We can ask less investment of the students, or we can give the students more for the same price.

Coursera, edX, and the traditional MOOC community take the former, providing content for a fraction of the cost because it can be delivered to so many people. Udacity, the Georgia Tech OMS, and other more expensive programs take the latter approach, reinvesting that money into creating higher-quality programs in the first place. Both these sides are critical. I don’t like living in a world where education is gated by such a massive monetary investment, and MOOC services are doing a world of good to reduce the barriers to education. At the same time, I love education itself, and I recognize that there are phenomenal things that the internet can do to improve education — but they come with a significant development cost.

Of course, this hasn’t actually answered the question: I’ve shown how openness and quality are distinct and often conflicting goals in online education, but can we accomplish both? Is it possible to create high-quality education that is also openly available for little to no monetary cost? It may be. At present, this is in some ways what the Georgia Tech OMS is doing: nine Georgia Tech courses are available for free to the world, and they are infused with a more significant initial investment that pays significant dividends in the quality of the instruction. This is accomplished because, in some ways, this free offering is “subsidized” by the students taking the actual Masters. This model is incomplete, however, as there is still valuable education locked within the for-cost program. OMS students are not paying for the videos; they are paying for the access to professors and TAs, the access to projects and assignments, and the ultimate “verified certificate”: the Masters diploma at the end of the program. However, this direction at least illustrates that it may be possible to use one offering in service of the other and improve both openness and quality at the same time. For now, however, I regard the two as distinct, exclusive, and desirable goals.

What’s the difference between online learning and distance learning?

At the Georgia Tech OMSCS, we talk a lot about how our program is the first of its kind. The homepage for the OMSCS states, “the first accredited Master of Science in Computer Science that students can earn exclusively through the Massive Open Online Course (MOOC) delivery format and for a fraction of the cost of traditional, on-campus programs.”

However, that description might not be entirely accurate. Georgia Tech OMS courses are neither massive (although they’re getting there) nor open (admission is still required). At the same time, the OMSCS is far from the only online Masters offered at Georgia Tech. So is all the hype just hype?

In my opinion, this speaks to the difference between distance learning and online learning, and the difference is critical. Distance learning has been around for ages through correspondence programs and other similar structures. Some of them are very good. Most online Masters programs today are simple extensions of distance learning programs. In previous years, one would receive course materials in the mail, and mail completed schoolwork back; now, students receive course materials over the internet, and upload completed assignments back. The internet makes distance learning easier, and at times can improve the experience through features like forums, but it does not inherently fundamentally change its structure.

The majority of online Masters programs are distance learning programs of this kind. With Georgia Tech’s online Masters programs, students in the distance learning sections view live or filmed lectures, upload the same assignments, and are graded by the same TAs. The only major difference is geographic: rather than being physically in the room of the lecture, the students are distributed. This is, in my mind, the heart of the distinction between distance learning and online learning: distance learning as nearly as possible identically recreates the in-person process. It may use the internet to do so, but the fundamental structure between distance learning and in-person learning remains the same.

Online learning, on the other hand, aims to leverage the internet not to duplicate the in-person experience, but rather to improve it. Improvement, of course, can come in many ways. Online education can be developed to reduce costs by leveraging MOOC principles, and in fact, this is one of the general guiding principles of the OMSCS: leveraging the internet to deliver an experience that is just as good as the in-person experience at a fraction of the cost. Online learning does not stop there, though. Automated feedback, communities of practice, and several other pedagogical techniques find unique places in the online medium. I’ve talked about a few of these unique benefits in the past, like the ability to transfer course ownership to the students and the natural emphasis on positive activity rather than negative, and I believe we’re only scratching the surface of the ways in which online education can actually improve on the in-person classroom experience.

I, of course, can be accused of bias in that, as an instructor and developer of the Georgia Tech OMS, I want to see it succeed. However, the inverse is true: I work on the Georgia Tech OMS because I believe it will succeed. I’m excited to work on it because while most programs out there are using the internet to improve on distance learning, the Georgia Tech OMS is about using the internet to create new and improved educational experiences altogether. Don’t get me wrong: there’s nothing wrong with distance learning, and it presents some very rich opportunities of its own. Distance learning is all about increasing access to the same quality education, and that is an incredibly important. I’m excited, though, to work on online learning and find ways to use the internet to make higher education more affordable, more accessible, and more effective.

So, if you’re ever asked why we ballyhoo the Georgia Tech OMS so much when online programs, even from highly reputable universities, are becoming common, the reason is that the OMS is about online learning, not distance learning. It’s very different, and it may lead to great things.

Beginning a Course: Developing Innovative Ideas for New Companies: The First Step in Entrepreneurship

One more course for now: University of Maryland’s Developing Innovative Ideas for New Companies: The First Step in Entrepreneurship, toward University of Maryland’s Entrepreneurship: Launching an Innovative Business specialization. It seems to be the immediately-available and bite-sized of Coursera’s new specializations, so it seems like a great place to start!

Should there exist accreditation for independent online courses?

One of the earliest takeaways from the handful of online courses I’m taking at the moment (Emerging Trends & Technologies in the Virtual K-12 Classroom, Learning How to Learn, and Astronomy: Exploring Space & Time, as well as a few days in Planet Earth and You) is that there exists a radical difference in the scope of different courses. Emerging Trends can be completed in a day if desired. Astronomy: Exploring Space & Time requires a much more significant time investment for the videos, but the interactive elements are relatively light, restricted to short quizzes and writing assignments. Planet Earth and You was much more significant, and decently approximated the amount of time I recall dedicating to traditional on-campus courses.

On the one hand, this is fantastic. Online learning has previously had the strength of not having to arbitrarily fit lessons to pre-determined time slot: if a topic takes more than a class period to instruct, it’s not necessary to arbitrarily break it up halfway through, and if it takes less than a class period to instruct, it’s not necessary to pad it out or randomly combine it with another topic. These courses reflect how that same idea can be expanded to an entire course. Not every course needs to be a three-hours-per-week 16-weeks-per-semester course. If a topic can be learned in 10 to 20 hours total (as Emerging Trends’ “2-4 hours per week, 5 weeks of study” guideline indicates), then let it be learned in that time frame.

On the other side, however, what does that say about how the world interprets these courses? A Bachelors is a Bachelors, a Masters is a Masters, and there exists some general understanding of what those degrees mean. The school from which a degree came has some influence, but universities have spent decades of time building up reputations to differentiate a Stanford Bachelors from a Samford Bachelors. Moreover, there are around 2500 four-year universities in the United States, and while that’s a large number, it isn’t intractable as far as developing an understanding of the different equivalence classes of universities.

There are currently around 1500 Udacity, Coursera, and edX courses combined, just to take three of the biggest organizations as examples. In the three years since these three organizations launched, they’ve developed almost as many courses as there are four-year institutions in the country. If the scope, depth, and rigor of individual open courses on these platforms is going to vary this much, how then will the world learn to interpret what it means to have a credential or certificate from these courses?

To put myself in the position of an employer, if a prospective employee had a verified course certificate from Coursera on their resume that I had not yet heard of, that would presently be somewhat meaningless to me; this is not because the certificate has no value, but just because I have no hope of knowing the certificate’s value at a glance, nor is it feasible to maintain a comprehensive knowledge of all the open courses I might see. This is a tremendous challenge to the value of these programs. Paying for a certified certificate is, for many, based on the belief that the ability to prove you completed a course is powerful. But if the people to whom you would offer that proof have no knowledge of what kind of knowledge and achievement that certificate represents, it remains somewhat meaningless.

This discussion comes dangerously close to the general discussion of accreditation. How does an employer know a certain college degree is valuable? Because it has been accredited by an independent organization. That knowledge of the program’s value and rigor has been offloaded to an external group for assessment. Just like a bank checking with a credit agency before deciding whether to give a person a loan, so also a business implicitly checks with an accreditation group before offering a graduate a job.

I wouldn’t argue that we need accreditation in the traditional sense for online courses; after all, many courses that wouldn’t pass a pass/fail accreditation process are nonetheless very valuable, even if they don’t necessarily demonstrate anything reliable about the students themselves. Based on my first impressions, I can’t say that hearing that a teacher has taken Emerging Trends & Technologies would mean much to my impression of them, but there is still lots they may have learned in the course. Rather, I feel what might be necessary is just a somewhat standardized classification system. Just like on-campus classes are assigned a number of credit hours based on the amount of work they require, so also online classes could be assigned a number or classification of virtual credits based on the rigor, reliability, and scope of the course itself. That, in turn, might lead to even greater programs: instead of a single university building up a Coursera specialization, it could instead be assembled from multiple universities’ courses on the basis of their virtual credits.

But apart from the solution, I feel the problem nonetheless exists and is waiting to be addressed: accreditation fulfills a function in traditional education; should online education have something to fulfill the same function?

Beginning a Course: Emerging Trends & Technologies in the Virtual K-12 Classroom

I’ll also be exploring a Coursera specialization in the Virtual Teacher Program from University of California-Irvine. Toward that end, I’ve also started the course Emerging Trends & Technologies in the Virtual K-12 Classroom, the second course in the program but the first one offered. Thoughts forthcoming!

Beginning a Course: Learning How to Learn

On the recommendation of a friend and because the course is available on-demand, I’ve started a second course as well: Learning How to Learn, from Barbara Oakley and Terrence Sejnowski at UC-San Diego. I commented in my previous blog post about Astronomy: Exploring Space & Time that gated, synchronous courses introduce a lot of pragmatic benefits but pedagogical drawbacks; I’ll be interested to see how an ungated course works.

As with Astronomy: Exploring Space & Time, I’ll be doing the Verified Certificate to get a look at the paid experience.

MOOCs and online agendas

During my first week in one of the Coursera MOOCs I’m taking, University of Arizona’s Astronomy: Exploring Space and Time, I observed something rather interesting in the community portion of the course. At first I was going to simply include it in my first week impressions, but it brings up a bigger issue that I think is worth examining more thoroughly.

I don’t want to go into the details of it, but early in the course, a thread sprung up started by someone claiming the earth is only 6,000 years old. Obviously, this claim has no place in a science class. While the claim was brought informally by saying, effectively, “Here’s what I think, how about everyone else?”, the responses that followed took, in my opinion, an evangelical turn.

The broader issue at play is this: when we teach a course in person, we tend to assume that the majority of people in the class are students whose primary “agenda” for the class is to either learn the material or get a good grade. In my ten years in academia, I can’t think of an instance of ever worrying about whether additional agendas were at play. This is largely because of the major obstacle to participation in an in-person college degree program: you’re paying a ton of money and dedicating an enormous amount of time, which would deter people with more subtle agendas.

In Coursera and other MOOCs, however, anyone can join, and for free. What is to deter competing agendas from entering the fray? In these MOOCs, there will be people who sign up and participate in these courses with agendas rather than to learn from the course’s stated purpose. There will be churches that consider it an act of evangelism to try to educate these people studying the origins of the universe on the earth’s “true” origin. There will be special interest groups that see classes on climate change or other environmental issues as opportunities to change the conversation in their favor or sow doubt when no doubt ought to exist. There will be political organizations wanting to present a biased view of history that is favorable to their platform. These MOOCs risk becoming battlegrounds rather than learning opportunities. We see this already on other social media platforms, where teams of people are paid by particular groups to try to sway the conversation in one way or the other; there is no obstacle that I can see to prevent open online courses from being subjected to the same kind of astroturfing.

When we think of college courses (or at least when I do), we think of them as opportunities for groups of learners to explore a topic together, typically guided by an expert in the field. When people are entering these courses with their own agendas that directly conflict with the material of the course, that opportunity is compromised.

So what do we do about this? I’m not sure. One could argue in favor of a heavy-handed approach, that anyone arguing for pseudoscience or false science in a college science course is simply kicked out; but the targets of that heavyhandedness would likely take this as a sign that science fears their alternate viewpoints and would sooner silence them than engage them in discussion. Those political groups would insist that it is we scientists who are concealing an agenda in our MOOCs. That is, of course, not the case; but an introductory science course is not the place to get into debates with people that have no intention of ever changing their viewpoint. Complicating matters further, though, is that for some of these people, their participation in these MOOCs is earnest: for many, this may be the first time they’re exposed to the scientific process, and they are willing to learn, and if we merely dismiss them as another fundamentalist with an agenda we miss the opportunity to introduce them to a broader view of the world.

So, a heavyhanded response is likely not the best option. Do we engage them in authentic discussion? That is a risky endeavor. For one, engaging in a discussion with completely unsubstantiated viewpoints risks giving them undeserved legitimacy. For another, any authentic discussion depends on both parties discussing on the same grounds; if we disagree on fundamental truths, then we may not ever meet a resolution. If we debate with someone who does not understand or believe the nature of evidence and theory-building, there is no shared understanding that can arise. It is a waste of our time and a needless distraction for the class.

If both shutting them down and engaging them in discussion are undesirable, what then can we do? Right now, my only idea is a third option: attempt to prevent them from entering the class in the first place. Of course, we can’t and shouldn’t do “interviews” for admission to a class for the same reasons stated above, but there are other ways that we can limit the participation of people with little desire to actually authentically learn. The first to come to mind for me is to introduce a barrier to entry. Attach a cost to pushing for your agenda. When anyone can “walk” in and start spouting off with their views as if they are as qualified as the professor, it is unsurprising that we get people promoting their views rather than engaging with course content. If a cost is associated with doing so, I would argue you limit the number of people who enter with an agenda rather than the desire to learn. Of course, this introduces the side effect of potentially preventing authentic learners from joining the class as well; however, we know that paying students are already far, far more likely to complete courses anyway. Depending on how you explain this data, this would mean that either those students who no longer join when there exists a price were not likely to complete the course anyway, or that forcing those students to pay will increase their likelihood of completing the course. Either option is desirable, though it is up for debate whether those would be the actual effects.

This isn’t all or nothing, of course. One could imagine a hybrid model where the payment associated simply gets you access to the community features, but that the content is available for free. This would preserve Coursera’s MOOCs’ desired openness while also attempting to safeguard its courses from outside influences. And, as a happy side effect, the money raised through this mechanism could go toward funding the classroom experience, paying for graders, compensating professors for their time, and building up the platform. I know I’d be very curious to see how the community for this class operated if I had the option of only interacting with those students also paying to participate in the course.

Intelligent tutoring online

My background is in intelligent tutoring systems. In intelligent tutoring systems, an artificial intelligence agent typically monitors student performance and reacts accordingly, giving feedback or support where necessary.

One of the interesting things about developing an online class is that because students are already engaging with a software system, the infrastructure and context necessary for an intelligent tutoring system are already present. What’s more, not only are they present, but they’re directly integrated into the context of the lesson. Whereas oftentimes intelligent tutoring systems are separate activities that complement a previously-received lecture, putting the learning online from the get-go allows us to integrate intelligent tutoring directly into the context of the lesson.

This development is in its infancy as far as I’m concerned, but I wanted to explain one way in which we use this in our Georgia Tech OMS class. Throughout the course, we have 125 interactive exercises each equipped with an AI agent – which I’ve taken to calling a ‘nanotutor’ to reflect the tiny scope of the skills that these agents teach – that gives students feedback on their latest responses. Let’s walk through an example of an exercise.

Continue reading Intelligent tutoring online