May 23, 2018

What UBC has learned about doing MOOCs

Coursera certificate 2

Engle, W. (2104) UBC MOOC Pilot: Design and Delivery Vancouver BC: University of British Columbia

The University of British Columbia, a premier public research university in Canada, successfully delivered five MOOCs in the spring and summer of 2013, using the Coursera platform. This report is an evaluation of the experience.

The report is particularly valuable because it provides details of course development and delivery, including media used and costs. Also UBC has been developing online courses for credit for almost 20 years, so it is interesting to see how this has impacted on the design of their MOOCs.


1. Game Theory I: K. Leyton Brown (UBC); M. Jackson and Y.Shoham (Stanford University)

2. Game Theory II: K. Leyton Brown (UBC); M. Jackson and Y.Shoham (Stanford University)

3. Useful Genetics: R. Redfield, UBC

4. Climate Literacy: S. Harris and S. Burch, UBC

5. Introduction to Systematic Program Design: G. Kizcales, UBC

In terms of comparability I’m going to treat Game Theory I and II as one MOOC, as combined they were about the same length as the other MOOCs (between 8-12 weeks)

Basic statistics

330,150 signed up (82,500 on average per course)

164,935 logged in at least once (41,000 per course)

12,031 took final exam (3,000 per course)

8,174 earned course certificate (2,000 per course)

60-70% already had a post-secondary degree

30-40% were North American, with participants from nearly every country in the world.

Course development

None of the instructors had taught an online course before, but were supported by instructional designers, media development staff, and academic assistants (graduate and undergraduate students).

One major difference between UBC MOOCs and its online credit courses (which are primarily LMS-based) was the extensive use of video, the main component of the MOOC pilot courses.

Video production

305 videos constituting a total of 65 hours were produced. Each MOOC used a different method of production:

  • Intensive studio (Climate Literacy)
  • Hybrid studio plus instructor desktop (Systematic Program Design)
  • Light studio production (Game Theory I and II)
  • Instructor desktop (Useful Genetics)

Web pages

All the MOOCs except Games Theory also included weekly modules as HTML-based web pages, which is a variation of the Coursera design default model. Altogether 98 HTML module pages were developed. The weekly modules were used to provide guidance to students on learning goals, amount of work expected, an overview of activities, and additional quiz or assignment help. (All standard practice in UBC’s LMS-based credit courses.)


1,049 quiz questions were developed, of which just over half were graded.

There were four peer assessments in total across all the MOOCs.

Course delivery

As well as the faculty member responsible for each MOOC, graduate and undergraduate academic assistants were a crucial component of all courses, with the following responsibilities:

  • directly assisting learners
  • troubleshooting technical problems
  • conducting quality assurance activities

There was very little one-on-one interaction between the main instructor and learners, but academic assistants monitored and moderated the online forum discussions.


As always, costing is a difficult exercise. Appendix B of the report gives a pilot total of $217,657, but this excludes academic assistance or, perhaps the most significant cost, instructor time.

Working from the video production costs ($95,350) and the proportion of costs (44%) devoted to video production in Figure 1 in the report, I estimate the direct cost at $216,700, or approximately $54,000 per MOOC, excluding faculty time and co-ordination support, but including academic assistance.

However, the range of cost is almost as important. The video production costs for Climate Literacy, which used intensive studio production, were more than six times the video production costs of Systematic Program Design (hybrid studio + desktop).


  • the UBC instructors are using their MOOC materials in their own on-campus, for-credit classes in a flipped classroom model
  • courses are left open and active on Coursera for self-paced learning
  • porting of video materials as open access YouTube videos
  • two courses (Climate Literacy and Useful Genetics) added Creative Commons licenses for re-use


  • copyright clearance (Coursera owns the copyright so third party copyright needs to be cleared)
  • higher than expected time demands on all involved
  • iterative upgrades to the Coursera platform
  • partner relationship management (UBC + Coursera + Stanford University) was time-consuming.
  • training and managing academic assistants, especially time management
  • the Coursera platform limited instructors’ ability to develop desired course activities
  • Coursera’s peer assessment functionality in particular was limiting


  • UBC’s prior experience in credit-based online learning led to better-designed, more interactive and more engaging MOOCs
  • learners always responded positively to instructor ‘presence’ in forums or course announcements
  • MOOC students were motivated by grades
  • MOOC students were willing to critically engage in critiquing instructors’ expertise and teaching
  • open publishing via MOOCs is a strong motivator for instructors
  • MOOCs require significant investment.


All the MOOCs received positive feedback and comments from students. UBC was able to gain direct experience in and knowledge of MOOCs and look at how this might inform both their for-credit on-campus and online teaching. UBC was also able to bring its experience in for-credit online learning to strengthening the design of MOOCs. Lastly it was able to make much more widely known the quality of UBC instructors and course materials.


First, congratulations to UBC for

  • experimenting with MOOCs
  • conducting the evaluation
  • making the report publicly available.

It is clear from the comments of participants in the appendices that at least some of the participants (we don’t know how many) were very pleased with the courses. As usual though with evaluation reports on MOOCs, there is no assessment of learning other than the end of course quiz-based tests. I don’t care too much about completion rates, but some measurement of student satisfaction would have been helpful.

It is also significant that UBC has now decided to move from Coursera to edX as its platform for MOOCs. edX, which is open source and allows partners to modify and adapt the platform, provides the flexibility that Coursera lacked, despite its many iterative ‘improvements’.

This also demonstrates the hubris of MOOC platform developers in ignoring best design principles in online learning when they designed their platforms. It is clear that UBC designers were able to improve the design of their MOOCs by drawing on prior for-credit online experience, but also that the MOOC platforms are still very limited in enabling the kind of learning activities that lead to student engagement and success.

The UBC report also highlighted the importance (and cost) of providing some form of learner support in course delivery. The use of academic assistants in particular clearly made the MOOCs more interactive and engaging, as well as limited but effective interventions from the instructors themselves, once again supported by (and confirming) prior research on the importance of instructor presence for successful for-credit online learning.

I very much appreciate the cost data provided by UBC, and the breakdown of production and delivery costs is extremely valuable, but I have to challenge the idea of providing any costs that exclude the time of the instructors. This is by far the largest and most important cost in MOOCs and the notion that MOOCs are free of instructor cost is to fly in the face of any respectable form of economics.

It is clear that MOOCs are more expensive to date per hour of study time than LMS-based for-credit online courses. We still do not have enough data to give a precise figure, and in any case, as the UBC study shows, cost is very much a factor of design. However, even without instructors costs, the UBC MOOCs at $54,000 each for between 8-12 weeks are already more than the average cost of a 13 week for-credit LMS-based online course, including instructor time.

This is partly due to the increased instructor time in preparation/production, but also to the higher cost of video production.  I am not against the use of video in principle, but it must add value. Using it for content transmission when this can be done so much more cheaply textually and/or by audio is a waste of the medium’s potential (although perhaps more motivating for the instructor).

More importantly, every institution contemplating MOOCs needs to do a cost-benefit exercise. Is it better to invest in MOOCs or credit-based online learning or both? If MOOCs are more expensive, what are the added benefits they provide and does this more than make up for not only the extra cost, but the lost opportunity of investing in (more) credit-based online learning or other forms of campus-based learning? I know what my answer would be.


MOOCs, specializations, and continuing education

Vanderbilit and University of maryland will be offering a MOOC specialization in designing  Android apps

Vanderbilit and University of Maryland will be offering a MOOC specialization in designing Android apps

Academic Partnerships (2014) Academic Partnerships Launches New Online Global Specializations Credential Dallas, January 21

Gannes, L. (2104) Coursera Offers the Equivalent of a MOOC Major: Specialization Certificates Re/Code, January 21

It must be more than a co-incidence that these two completely separate organizations announced new ‘specialization’ certificates on the same day. First, a little background.

Coursera certificates

Coursera is planning to offer certificates for students who take a set combination of MOOCs and pass the assessment. The minimum number of MOOCs would be three, with other certificates requiring up to eight MOOCs. The certificates will be awarded by ‘leading universities.’ One of the first specializations open for enrollment is from Vanderbilt and the University of Maryland on making Android apps.  A Coursera specialization certificate will require students to verify their identity and pay on a per-course basis, usually $49 per course.

Academic Partnerships

Most people will know about Coursera, but Academic Partnerships may be less well known, but is still a significant player in the higher education world of the USA. Its is a private company that ‘assists universities in converting their traditional degree programs and certificates into an online format, recruits qualified students, and supports enrolled students through graduation’. It works particularly with prestigious U.S. institutions that often were slow into credit-based online learning, or those that wish to keep the online learning activity at somewhat arms-length from their campus activities, but usually to increase enrollments and/or revenues.

Academic Partnership’s new initiative is on the global marketing of specialization certificates from prestigious U.S. universities, ‘to help partner universities capitalize on the globalization of higher education….Specializations consist of three progressive certificates that are offered in multiple languages and can be earned in nine months….AP’s partner universities outside of the United States, meanwhile, will serve as host universities for the Specializations….We believe that our Specializations initiative, which we originated and are launching with partner universities, will significantly increase post-secondary enrollment around the world, resulting in untold benefits for citizens everywhere, while simultaneously addressing the financial challenges faced by U.S. universities.’


These seem like sensible moves, moving MOOCs and other ‘open’ online courses much more clearly into the continuing education niche, and internationalizing them. The idea of opening up online courses to international markets of course is not new. The University of British Columbia launched an ‘open’ (but not free) online certificate program on distributed learning as early as 1996, which developed into an international Master in Educational Technology in 2003 (which is still very successful). The first enrollments for UBC’s certificate program came from over 30 different countries. There is clearly a large international market for online courses and programs from prestigious North American institutions, and Australia universities for many years have had a highly profitable international online learning presence.

Questions still remain though. One obvious one is about the transferability of credit from specializations: will students obtaining certificates be able to transfer these into regular credit programs, online or on campus, at North American or local universities? Will students, especially those overseas, be told this when they enroll?

The second challenge is the business model, at least for the MOOC initiative. It is not so much the MOOCs or courses themselves – they may be offered free by the institutions. But how will this impact on their Continuing Education divisions, who often exist now mainly to bring in extra revenues for the university? Many universities – at least here in Canada – have extensive online certificate programs which bring in a large profit for the university. One wonders why institutions would ‘give away’ this highly lucrative market in order to provide free, open courses. Or maybe MOOCs will destroy the current continuing education model for online courses, which will send a chill through many universities’ CE divisions.

For the Academic Partnership model, the success will depend on the added value that Academic Partnerships can bring to the university in international marketing and recruitment. It will be interesting to see how they price these certificates in the different international markets. There is also a strong Hawthorne effect here. As institutions in other countries begin to build their own MOOCs and online credit programs, the market starts to drop for programs from other countries, so timing is everything.

One thing that both initiatives though should be aware of – and this is free advice – from my experience in internationalizing online courses is that one needs to be aware of this from the beginning, ensuring the materials are appropriate for multicultural use (which goes way beyond direct translation), and above all, that there is 24×7 online learner support available in some form or other that extends beyond answering simple technical or administrative questions. In particular, international students will want to know what they can do with these specialization certificates and which employers are likely to recognize them. Since this will vary greatly from country to country, this is no simple or low-cost task. This is the major challenge in internationalizing MOOCs, and will require a very strong business model and excellent partners in the other countries for the Academic Partnership’s initiative to succeed.

Lastly, if I was a Director of Continuing Education or International Education, I would not be sitting back waiting for these initiatives to happen, but would be developing a business plan to go out and compete directly for international students and lifelong learners through online learning.


Online learning in California generates controversy

© Robert Holmes, CalTour, 2012

Booker, E. (2013) Online education policy draws fire in California Information Week: Education, January 11

This report of a one-day conference at UCLA: Rebooting Higher Education: Leveraging Innovations in Online Education to Improve Cost Effectiveness and Increase Quality,’ resulted in ‘heated critiques about the rationale for the state’s policy push into the area.’

Some of the points raised:

  • the Governor has created a fund for open textbooks and an open digital library for the state’s public post-secondary system (a model being imitated in British Columbia)
  • criticism from faculty associations that the focus on MOOCs doesn’t address the needs of undergraduate students, and that the rush into MOOCs wasn’t being accompanied by research or evaluation of their effectiveness
  • criticisms from the CEO of Straighterline, an online service offering introductory courses, that public universities were monopolies subsidized by taxpayers that were keeping out cheaper commercial providers
  • Daphne Koller of Coursera argued that MOOCs are ushering in a “brand new pedagogy” and provide important keys about effective teaching and educational design by using Google-style rapid testing resulting in daily improvements to the web site
  • the number of full-time-equivalent online students in the state’s community college network has doubled since 2005-2006, and now constitute 11% of all course enrollments.

The conference was sponsored by the 20 Million Minds Foundation, a nonprofit organization dedicated to lowering the cost of higher education.


California appears likely to be a key battle ground regarding the role of the private sector and online learning in post-secondary education. Public colleges and universities have seen steep cuts in their funding over recent years and the state’s finances do not look to be any healthier for at least the next two or three years.

There are in fact several strands here that are getting mixed up and perhaps should be considered separately:

  • the commitment of voters in California to a publicly funded post-secondary education system. The current financial difficulties of the state are a consequence of the ‘tax revolt’ by Californians over the last 25 years resulting from  proposition 13 in 1978 that outlawed any property tax increases for ever in California. If you don’t pay taxes, you can’t afford public services. There seems to be a gradual understanding of this, because the governor was able to get some increase in state taxes through proposition 30 that was passed in a state ballot in November.
  • ‘traditional’ for-credit online learning is already well-established in the public post-secondary system in California, with somewhere between 11-15% of all for-credit course enrollments now being online. This ‘natural’ growth is likely to continue, with or without the influence of MOOCs, although the publicity around MOOCs may (or may not) facilitate this
  • MOOCs may have some relevance for for-credit programming, but that is not their current purpose. There is still a great deal of work to be done before they come close to being an adequate substitute for current for-credit online courses, and indeed that may turn out to be the least valuable road for MOOCs to travel. Whatever road they do follow, they will need to demonstrate their effectiveness through independent evaluation of what students actually learn
  • effective teaching is not the same as effective advertising, so assuming that Google-style ad testing will result in educational breakthroughs is just dreaming. To sort out the signal from the noise in analyzing big data, you need hypotheses or theories, so you know what kind of data to collect, and how to interpret it when it is collected. The theories for learning are almost certainly different than the theories for selling or marketing. It’s about time Coursera recognized that we do know a good deal about what leads to effective online learning, and this has not been applied to most Coursera MOOCs to date. The sooner this is done, the more effective they are likely to be, and you don’t needs tons of data for this, although it could be helpful once these approaches are applied, to further refine our knowledge in this area
  • despite all this, the costs of post-secondary education in the USA in particular, and many other OECD countries, including Canada, are too high and as a result are becoming an increasing barrier to access, so we do need more experimentation, innovation, research and evaluation. However this needs to be more thoughtful than just jumping off a cliff and hoping there’s deep water underneath
  • lastly, commercial providers can supply online learning offerings more cheaply than public organizations, because the public organizations are not just offering rote learning, but at least aiming for the development of critical thinking skills, originality, communications skills, networking, and above all the generation of new knowledge and innovation which is partly cross-subsidized by funds coming in for teaching (whether that should be the case is another matter). In other words, commercial online providers go for the low-hanging fruit, and leave the public sector to pick up the rest. As in the stock market, the commercial providers want to take the profits, and leave the losses to the taxpayer.

California is going to be one of the major focal points for these issues to be played out. It is one of the biggest public post-secondary systems in the USA, and it will be a marker for the rest of the country in this struggle, which reflects perhaps the most important ideological issues in education for many years.  However, public thinking, especially in the USA, is often grossly over-simplified into two opposing positions. Unfortunately – or fortunately, I’m not sure which – online learning is caught right in the middle. Online learning in fact can be a weapon used by either side. We should be looking for a third road, where online learning helps public institutions to become more cost-effective, through fundamental changes that at the same time strengthen the good parts of the public system: equitable access, quality teaching, new knowledge production, and intellectual freedom.

Emerging business models for MOOCs

© Board of Innovation, 2012

Gilfus Education Group (2012) Coursera will profit from ‘free’ courses, competition heats up, Gilfus Education Group blog, undated, accessed November 12

Gilfus Education Group, an independent third party think tank out of Washington, DC, sets out eight ‘monetization strategies’ for MOOC providers such as Coursera, and three possible business models (Freemium, Value-added, and Corporate Social Responsibility).

The blog also provides links to three contracts publicly posted by institutions with Coursera contracts (Michigan, Illinois and Toronto), where it appears that the institutions get between 6-15% of revenues (presumably just from their own MOOCs), while Coursera gets 20% of profits, after all expenses are paid (presumably the venture capitalists/investors get the other 80% of profits).

Gilfus comments that: ‘Sustainability is not assured with the MOOC organizations until they can find an economic model that rewards all stakeholders. It might in fact be easier for existing LMS companies to pursue these models as the technology already exists on campus. Simply put companies like Blackboard, Desire2Learn, Moodlerooms and others are quickly devising strategies to develop high quality courses in addition to their current capabilities.


Indeed, all a company like Desire2Learn or Blackboard needs to do is set up a portal of online courses available for free from existing universities already using their LMS for fully online credit course delivery (with the institution’s agreement of course). Instructure’s Canvas Network already does this and already has ‘open’ courses from several universities (University of Central Florida, Ball State, Brown, Utah State and the University of Utah are just examples)

But even that isn’t really necessary. If the institutions are already offering online courses for credit, the institutions could handle the rest, making available very quickly, for free, a vast range of  quality online courses, but not for credit, so why would they sign up with Coursera, or even an existing LMS company?

The universities already with credit online courses just need to tweek their registration system to distinguish between free, not-for-credit students and tuition-paid credit students. The non-credit version is unlimited in admission, but non-credit students get no tutorial support or individualized assessment (as in current MOOCs), whereas the credit students get the full works. This is an extremely low-cost way of opening up all your teaching to the general public. The only advantage of using a third-party portal such as Desire2Learn or Coursera would be to have one central point with a vast range of free courses from many different institutions (such as iTunesU). However, there would be no direct profit to the universities from this approach.

But just in case you were under the impression that it is openness that these institutions are interested in, I need to disabuse you. You have to wonder why publicly-funded universities in particular (such as the University of Toronto) want to join Coursera if it’s not to make money. Isn’t there though some conflict of interest here, or am I missing something? (Perhaps we should change the name to MOCCs – Massive Online Corporate Courses).

What elite institutions such as the University of Toronto are doing in fact is trying to get into online learning on the cheap, and for profit, after years of turning their noses up at it. But now Stanford, MIT and Harvard are doing it, it’s OK. Shame on them.

Thanks to Richard Pinent, University of Ottawa, for directing me to the Gilfus Education Group blog post.

MOOCs move into credit-based higher education

Kolowich, S. (2012) MOOCs for credit, Inside Higher Education, October 29

This article reports on an interesting deal signed between Coursera and Antioch University. Antioch is a well-established private university with campuses in four different states, with around 4,000 mainly adult students. Under this deal:

  • Antioch University will pay Coursera for the rights to offer MOOCs from 33 universities for credit as part of its third year undergraduate program, focusing particularly on students transferring in from community colleges (the fourth year will be on campus)
  • Antioch will charge a lower tuition fee for these courses (closer to community college fee levels than public universities’)
  • Antioch will assign a faculty member to provide learner support for Antioch-registered students in each MOOC-based course, with about 20 students per instructor (as for on-campus classes)
  • Students will take an exam at the end of each course for credit from Antioch
  • Coursera will pay rights to the universities contributing MOOC courses to Antioch.

Inside Higher Education reports:

For Coursera, which is still building its MOOC empire with venture capital, the Antioch deal is a first step toward developing a product that it can sell to colleges: “self-contained” online course platforms, complete with built-in content and assessment infrastructure.

“It’s an LMS [learning management system] that’s wrapped around a very high-quality course,” says Koller, the co-founder. “It’s not just the box, it’s a course in a box.”


Although this is starting as a pilot, it provides a possible means to reduce the costs of US university education, while still providing learner support and credits.

It is also interesting that Antioch feels it will gain prestige from using MOOCs from other universities.

However, to date the model requires the faculty member to take on the learner support in addition to their regular teaching load. My own research suggests that over the long term, learner support costs are two to three times the costs for development (which is effectively what Antioch is paying Coursera for), so whether the business model makes sense for Antioch remains to be seen. It is though a low cost way for it to get into online learning.

Furthermore, learner support costs could be greatly reduced if the course content was developed following best instructional design principles for independent online learners, rather than canned lectures. So perhaps in the long run, quality in design may become an important ‘selling’ factor in MOOCs, more important perhaps than the name of the institution canning the lectures. I sincerely hope so, but then I’m just a silly old romantic.

In this case, the courses from Antioch will not be massive, will not be open, and will not be free. So when is a MOOC not a MOOC? Nevertheless it is an interesting model, and could provide wins for Antioch, the institutions providing MOOCs, and Coursera. Whether it’s a win for the students though remains to be seen.