May 23, 2015

Why MOOCs are only part of the answer for higher education

Listen with webReader
Formal education is the necessary launchpad for successful MOOCs

Formal education is the necessary launchpad for successful MOOCs

OK, except for the next post, which will be a list of publications on MOOCs for graduate students studying the topic, and a scenario for a ‘good’ MOOC, this will be my last post on MOOCs for a while.

This is the conclusion to my chapter on MOOCs for my open textbook, ‘Teaching in a Digital Age‘. This whole chapter is now published here. There will be a scenario illustrating what I define as a ‘good’ MOOC to go with this conclusion. Here is the extract:

The importance of context and design

I am frequently categorised as a major critic of MOOCs, which is somewhat surprising since I have been a longtime advocate of online learning. In fact I do believe MOOCs are an important development, and under certain circumstances they can be of tremendous value in education.

But as always in education, context is important. There is not one but many different markets and needs for education. A student leaving high school at eighteen has very different needs and will want to learn in a very different context from a 35 year old employed engineer with a family who needs some management education. Similarly a 65 year old man struggling to cope with his wife’s early onset of Alzheimers and desperate for help is in a totally different situation to either the high school student or the engineer. When designing educational programs, it has to be horses for courses. There is no single silver bullet or solution for every one of these various contexts.

Secondly, as with all forms of education, how MOOCs are designed matters a great deal. If they are designed inappropriately, in the sense of not developing the knowledge and skills needed by a particular learner in a particular context, then they have little or no value for that learner. However, designed differently and a MOOC may well meet that learner’s needs.

The potential of cMOOCs

So let me be more specific. cMOOCs have the most potential, because lifelong learning will become increasingly important, and the power of bringing a mix of already well educated and knowledgeable people from around the world to work with other committed and enthusiastic learners on common problems or areas of interest could truly revolutionise not just education, but the world in general.

However, cMOOCs at present are unable to do this, because they lack organisation and do not apply what is already known about how online groups work best. Once we learn these lessons and apply them, though, cMOOCs can be a tremendous tool for tackling some of the great challenges we face in the areas of global health, climate change, civil rights, and other ‘good civil ventures.’  The beauty of a cMOOC is that they involve not just the people who have the will and the power to make changes, but cMOOCs give every participant the power to define and solve the problems being tackled.

But socially transformative MOOCs will almost certainly benefit from the resources of strong institutions to provide initial impetus, simple to use software, overall structure, organization and co-ordination within the MOOC, and some essential human resources for supporting the MOOC when running. At the same time, it does not have to be an educational institution. It could be a Public Health Authority, or a broadcasting organization, or an international charity, or a consortium of organisations with a common interest. Also, of course, we need to recognise the danger that even cMOOCs  could be manipulated by corporate or government  interests. Finally, I don’t see cMOOCs as being a replacement for formal education, but as a rocket that needs formal education as its launch pad.

The limitations of xMOOCs

The real threat of xMOOCs is to the very large face-to-face lecture classes found in many universities at the undergraduate level. MOOCs, at a cost of around $20-$50 a student, are a more effective way of replacing such lectures. They are more interactive and permanent so students can go over the materials many times. I have heard MOOC instructors argue that their MOOCs are better than their classroom lectures. They put more care and effort into them.

However, we should question why we are teaching in this way on campus. Content is now freely available anywhere on the Internet – including MOOCs. What is needed is information management: how to identify the knowledge you need, how to evaluate it, how to apply it. MOOCs do not do that. They pre-select and package the information. My big concern with xMOOCs is their limitation, as currently designed, for developing the higher order intellectual skills needed in a digital world. Unfortunately, xMOOCs are taking the least appropriate design model for developing 21st century skills from on-campus teaching,  and moving this inappropriate design model online. Just because the lectures come from elite universities does not necessarily mean that learners will develop high level intellectual skills, even though the content is of the highest quality. More importantly, with MOOCs, relatively few students succeed, in terms of assessment, and those that do are tested mainly on comprehension and limited application of knowledge.

We can and have done much better in terms of skills for a digital age with other pedagogical approaches on campus, such as problem- or inquiry-based learning, and with online learning using more constructivist approaches in online credit courses, but these alternative methods to lectures do not scale so easily. The interaction between an expert and a novice still remains critical for developing deep understanding, transformative learning resulting in the learner seeing the world differently, and for developing high levels of evidence-based critical thinking, evaluation of complex alternatives, and high level decision-making. Computer technology to date is extremely poor at enabling this kind of learning to develop. This is why credit-based classroom and online learning still aim to have a relatively low instructor:student ratio and still need to focus a great deal on interaction between instructor and students.

I have no problem however with xMOOCs as a form of continuing education or as a source of open educational materials that can be part of a broader educational offering. They can be a valuable supplement to campus-based education. It is when the claim is made that they can replace both conventional education or the current design of online credit programs when I become really concerned. As a form of continuing education, low completion rates and the lack of formal credit is not of great significance. However, completion rates and quality assessment DO matter if MOOCs are being seen as a substitute or a replacement for formal education, even classroom lectures.

Undermining the public higher education system?

The real danger is that if we are not vigilant, MOOCs will undermine what is admittedly an expensive public higher education system. If elite universities can deliver MOOCs for free, why do we need crappy state universities? The risk is a sharply divided two tier system, with a relatively small number of elite universities catering to the rich and privileged, and developing the knowledge and skills that will provide rich rewards, and the masses going to MOOC-delivered courses with state universities providing minimal and low cost learner support for such courses. This would be both a social and economic disaster, because it would fail to produce enough learners with the high-level skills that are going to be needed for good jobs in the the coming years – unless you believe that automation will remove all decently paid jobs except for a tiny elite (bring on the Hunger Games).

It should be noted that even for credit-based online programs, content accounts for less than 15 per cent of the total cost over five years; the main costs required to ensure high quality outcomes and high rates of completion are spent on learner support, providing the learning that matters most. The kind of MOOCs being promoted by politicians and the media fail spectacularly to do this. We do need to be careful that the open education movement in general, and MOOCs in particular, are not used as a stick by those in the United States and elsewhere who are deliberately trying to undermine public education for ideological and commercial reasons. Open content, OERs and MOOCs do not automatically lead to open access to high quality credentials for everyone. In the end, a well-funded public higher education system remains the best way to assure access to higher education for the majority of the population.

Having said that, there is enormous scope for improvements within that system. MOOCs, open education and new media offer promising ways to bring about some much needed improvements. However, that means building on what we already know from the use of credit based online learning, from prior experience in open and distance learning, and designing courses and programs in a variety of ways appropriate to the wide range of learning needs. MOOCs can be one important part of that environment, but not a replacement for other forms of educational provision that meet different needs.

Key Takeaways

1. MOOCs are forcing every higher education institution to think carefully both about its strategy for online teaching and its approach to open education.

2. MOOCs are not the only form of online learning or of open educational resources. It is important to look at the strengths and weaknesses of MOOCs within the overall context of online learning and open-ness.

3. There are considerable differences in the design of MOOCs, reflecting different purposes and philosophies.

4. MOOCs are at still a relatively early stage of maturity. As their strengths and weaknesses become clearer, and as experience in improving their design grows, they are likely to occupy a significant niche within the higher education learning environment.

5. There are still major structural limitations in MOOCs for developing deep or transformative learning, or for developing the high level knowledge and skills needed in a digital age.

6. MOOCs could well replace some forms of traditional teaching (such as large lecture classes). However, MOOCs are more likely to remain an important supplement or alternative to other conventional education methods. They are not on their own a solution to the high cost of higher education, although MOOCs are and will continue to be an important factor in forcing change.

7. Perhaps the greatest value of MOOCs in the future will be for providing a means for tackling large global problems through community action.

Next

I am now trying to finish Chapter 6, on design models. I will be writing about (a) personal learning environments and (b) flexible design models based on sound educational design principles.

As always, I welcome comments on either this final section on MOOCs, or on the Chapter as a whole. You can use either the comment page here or the one at the end of the Chapter.

Why the fuss about MOOCs? Political, social and economic drivers

Listen with webReader
Daphne Koller's TED talk on MOOCs (click to activate video)

Daphne Koller’s TED talk on MOOCs (click to activate video)

The end of MOOCs

This is the last part of my chapter on MOOCs for my online open textbook, Teaching in a Digital Age. In a series of prior posts, I have looked at the strengths and weaknesses of MOOCs. Here I summarise this section and look at why MOOCs have gained so much attention.

Brief summary of strengths and weaknesses of MOOCs

The main points of my analysis of the strengths and weaknesses of MOOCs can be summarised as follows:

Strengths

  • the main value proposition of MOOCs is that through the use of computer automation and/or peer-to-peer communication MOOCs can eliminate the very large variable costs in higher education associated with providing learner support and quality assessment
  • MOOCs, particularly xMOOCs, deliver high quality content from some of the world’s best universities for free to anyone with a computer and an Internet connection
  • MOOCs can be useful for opening access to high quality content, particularly in Third World countries, but to do so successfully will require a good deal of adaptation, and substantial investment in local support and partnerships
  • MOOCs are valuable for developing basic conceptual learning, and for creating large online communities of interest or practice
  • MOOCs are an extremely valuable form of lifelong learning and continuing education
  • MOOCs have forced conventional and especially elite institutions to reappraise their strategies towards online and open learning
  • institutions have been able to extend their brand and status by making public their expertise and excellence in certain academic areas

Weaknesses

  • the high registration numbers for MOOCs are misleading; less than half of registrants actively participate, and of these, only a small proportion successfully complete the course; nevertheless, absolute numbers of successful participants are still higher than for conventional courses
  • MOOCs are expensive to develop, and although commercial organisations offering MOOC platforms have opportunities for sustainable business models, it is difficult to see how publicly funded higher education institutions can develop sustainable business models for MOOCs
  • MOOCs tend to attract those with already a high level of education, rather than widen access
  • MOOCs so far have been limited in the ability to develop high level academic learning, or the high level intellectual skills needed in a knowledge based society
  • assessment of the higher levels of learning remains a challenge for MOOCs, to the extent that most MOOC providers will not recognise their own MOOCs for credit
  • MOOC materials may be limited by copyright or time restrictions for re-use as open educational resources

Why the fuss about MOOCs?

It can be seen from the previous section that the pros and cons of MOOCs are finely balanced. Given though the obvious questions about the value of MOOCs, and the fact that before MOOCs arrived, there had been substantial but quiet progress for over ten years in the use of online learning for undergraduate and graduate programs, you might be wondering why MOOCs have commanded so much media interest, and especially why a large number of government policy makers, economists, and computer scientists have become so ardently supportive of MOOCs, and why there has been such a strong, negative reaction, not only from many traditional university and college instructors, who are right to be threatened by some of the claims being made for MOOCs, but also from many professionals in online learning (see for instance, Bates, 2012; Daniel, 2012; Hill, 2012; Watters, 2013), who might be expected to be more supportive of MOOCs

It needs to be recognised that the discourse around MOOCs is not usually based on a cool, rational, evidence-based analysis of the pros and cons of MOOCs, but is more likely to be driven by emotion, self-interest, fear, or ignorance of what education is actually about. Thus it is important to explore the political, social and economic factors that have driven MOOC mania.

Massive, free and Made in America!

This is what I will call the intrinsic reason for MOOC mania. It is not surprising that, since the first MOOC from Stanford professors Andrew Ng and Daphne Koller attracted 270,000 sign-ups from around the world, since the course was free, and since it came from professors at one of the most prestigious private universities in the USA, the American media were all over it. It was big news in its own right, however you look at it, especially as courses from Sebastian Thrun, another Stanford professor, and others from MIT and Harvard followed shortly, with equally staggering numbers of participants.

It’s the Ivy Leagues!

Until MOOCs came along, the major Ivy League universities in the USA, such as Stanford, MIT, Harvard and UC Berkeley, as well as many of the most prestigious universities in Canada, such as the University of Toronto and McGill, and elsewhere, had largely ignored online learning in any form.

However, by 2011, online learning, in the form of for credit undergraduate and graduate courses, was making big inroads at many other, very respectable universities, such as Carnegie Mellon, Penn State, and the University of Maryland in the USA, and also in many of the top tier public universities in Canada and elsewhere, to the extent that almost one in three course enrolments in the USA were now in online courses. Furthermore, at least in Canada, the online courses were often getting good completion rates and matching on-campus courses for quality.

The Ivy League and other highly prestigious universities that had ignored online learning were beginning to look increasingly out of touch by 2011. By launching into MOOCs, these prestigious universities could jump to the head of the queue in terms of technology innovation, while at the same time protecting their selective and highly personal and high cost campus programs from direct contact with online learning. In other words, MOOCs gave these prestigious universities a safe sandbox in which to explore online learning, and the Ivy League universities gave credibility to MOOCs, and, indirectly, online learning as a whole.

It’s disruptive!

For years before 2011, various economists, philosophers and industrial gurus had been predicting that education was the next big area for disruptive change due to the march of new technologies (see for instance Lyotard, 1979; Tapscott, undated; Christensen and Eyring, 2011).

Online learning in credit courses though was being quietly absorbed into the mainstream of university teaching, through blended learning, without any signs of major disruption, but here with MOOCs was a massive change, providing evidence at long last in the education sector to support the theories of disruptive innovation.

It’s Silicon Valley!

It is no coincidence that the first MOOCs were all developed by entrepreneurial computer scientists. Ng and Koller very quickly went on to create Coursera as a private commercial company, followed shortly by Thrun, who created Udacity. Anant Agarwal, a computer scientist at MIT, went on to head up edX.

The first MOOCs were very typical of Silicon Valley start-ups: a bright idea (massive, open online courses with cloud-based, relatively simple software to handle the numbers), thrown out into the market to see how it might work, supported by more technology and ideas (in this case, learning analytics, automated marking, peer assessment) to deal with any snags or problems. Building a sustainable business model would come later, when some of the dust had settled.

As a result it is not surprising that almost all the early MOOCs completely ignored any pedagogical theory about best practices in teaching online, or any prior research on factors associated with success or failure in online learning. It is also not surprising as a result that a very low percentage of participants actually successfully complete MOOCs – there’s a lot of catching up still to do, but so far Coursera and to a lesser extent edX have continued to ignore educators and prior research in online learning. They would rather do their own research, even if it means re-inventing the wheel. The commercial MOOC platform providers though are beginning to work out a sustainable business model.

It’s the economy, stupid!

Of all the reasons for MOOC mania, Bill Clinton’s famous election slogan resonates most with me. It should be remembered that by 2011, the consequences of the disastrous financial collapse of 2008 were working their way through the economy, and particularly were impacting on the finances of state governments in the USA.

The recession meant that states were suddenly desperately short of tax revenues, and were unable to meet the financial demands of state higher education systems. For instance, California’s community college system, the nation’s largest, suffered about $809 million in state funding cuts between 2008-2012, resulting in a shortfall of 500,000 places in its campus-based colleges. Free MOOCs were seen as manna from heaven by the state governor, Jerry Brown.

One consequence of rapid cuts to government funding was a sharp spike in tuition fees, bringing the real cost of higher education sharply into focus. Tuition fees in the USA have increased by 7% per annum over the last 10 years, compared with an inflation rate of 4% per annum. Here at last was a possible way to rein in the high cost of higher education.

Now though the economy in the USA is picking up and revenues are flowing back into state coffers, and so the pressure for more radical solutions to the cost of higher education is beginning to ease. It will be interesting to see if MOOC mania continues as the economy grows, although the search for more cost-effective approaches to higher education is not going to disappear.

Don’t panic!

These are all very powerful drivers of MOOC mania, which makes it all the more important to try to be clear and cool headed about the strengths and weaknesses of MOOCs. The real test is whether MOOCs can help develop the knowledge and skills that learners need in a knowledge-based society. The answer of course is yes and no.

As a low-cost supplement to formal education, they can be quite valuable, but not as a complete replacement. They can at present teach conceptual learning, comprehension and in a narrow range of activities, application of knowledge. They can be useful for building communities of practice, where already well educated people or people with a deep, shared passion for a topic can learn from one another, another form of continuing education.

However, certainly to date, MOOCs have not been able to demonstrate that they can lead to transformative learning, deep intellectual understanding, evaluation of complex alternatives, and evidence-based decision-making, and without greater emphasis on expert-based learner support and more qualitative forms of assessment, they probably never will, at least without substantial increases in their costs.

At the end of the day, there is a choice between throwing more resources into MOOCs and hoping that some of their fundamental flaws can be overcome without too dramatic an increase in costs, or whether we would be better investing in other forms of online learning and educational technology that could lead to more cost-effective learning outcomes. I know where I would put my money, and it’s not into MOOCs.

Over to you

This will be my last contribution to the discussion of MOOCs for my book, so let’s have it!

1. Do you agree with the strengths and weaknesses of MOOCs that I have laid out? What would you add or remove or change?

2. What do you think of the drivers of MOOC mania? Are these accurate? Are there other, more important drivers of MOOC mania?

3. Do you even agree that there is a mania about MOOCs, or is their rapid expansion all perfectly understandable?

References

Bates, T. (2012) What’s right and what’s wrong about Coursera-style MOOCs, Online learning and distance education resources, August 5

Christensen, C. and Eyring, H. (2011), The Innovative University: Changing the DNA of Higher Education, New York, New York, USA: John Wiley & Sons,

Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility.Journal of Interactive Media in Education, Vol. 3

Hill, P. (2012) Four Barriers that MOOCs must overcome to build a sustainable model, e-Literate, July 24

Lyotard, J-J. (1979) La Condition postmoderne: rapport sur le savoir: Paris: Minuit

Tapscott, D. (undated) The transformation of education dontapscott.com

Watters, A. (2013) MOOC Mania: Debunking the hype around massive, open online courses The Digital Shift, 18 April

The strengths and weaknesses of MOOCs: Part 2: learning and assessment

Listen with webReader
Remote exam proctoring

Remote exam proctoring

The writing of my book, Teaching in a Digital Age, has been interrupted for nearly two weeks by my trip to England for the EDEN Research Workshop. As part of the first draft of the book, I have already published three posts on MOOCs:

In this post, I ask (and try to answer) what do participants learn from MOOCs, and I also evaluate their assessment methods.

What do students learn in MOOCs?

This is a much more difficult question to answer, because so little of the research to date (2014) has tried to answer this question. (One reason, as we shall see, is that assessment of learning in MOOCs remains a major challenge). There are at least two kinds of study: quantitative studies that seek to quantify learning gains; and qualitative studies that describe the experience of learners within MOOCs, which indirectly provide some insight into what they have learned.

At the time of writing, the most well conducted study of learning in MOOCs has been by Colvin et al. (2014), who investigated ‘conceptual learning’ in an MIT Introductory Physics MOOC. They compared learner performance not only between different sub-categories of learners within the MOOC, such as those with no physics or math background with those such as physic teachers who had considerable prior knowledge, but also with on-campus students taking the same curriculum in a traditional campus teaching format. In essence, the study found no significant differences in learning gains between or within the two types of teaching, but it should be noted that the on-campus students were students who had failed an earlier version of the course and were retaking it.

This research is a classic example of the no significant difference in comparative studies in educational technology; other variables, such as differences in the types of students, were as important as the mode of delivery. Also, this MOOC design represents a behaviourist-cognitivist approach to learning that places heavy emphasis on correct answers to conceptual questions. It doesn’t attempt to develop the skills needed in a digital age as identified in Chapter 1.

There have been far more studies of the experience of learners within MOOCs, particularly focusing on the discussions within MOOCs (see for instance, Kop, 2011). In general (although there are exceptions), discussions are unmonitored, and it is left to participants to make connections and respond to other students comments.

However, there are some strong criticisms of the effectiveness of the discussion element of MOOCs for developing the high-level conceptual analysis required for academic learning. To develop deep, conceptual learning, there is a need in most cases for intervention by a subject expert, to clarify misunderstandings or misconceptions, to provide accurate feedback,  to ensure that the criteria for academic learning, such as use of evidence, clarity of argument, etc., are being met, and to ensure the necessary input and guidance to seek deeper understanding (see Harasim, 2013).

Furthermore, the more massive the course, the more likely participants are to feel ‘overload, anxiety and a sense of loss’, if there is not some instructor intervention or structure imposed (Knox, 2014). Firmin et al. (2014) have shown that when there is some form of instructor ‘encouragement and support of student effort and engagement’, results improve for all participants in MOOCs. Without a structured role for subject experts, participants are faced with a wide variety of quality in terms of comments and feedback from other participants. There is again a great deal of research on the conditions necessary for the successful conduct of collaborative and co-operative group learning (see for instance, Dillenbourg, 1999, Lave and Wenger, 1991), and these findings certainly have not been generally applied to the management of MOOC discussions to date.

One counter argument is that at least cMOOCs develop a new form of learning based on networking and collaboration that is essentially different from academic learning, and MOOCs are thus more appropriate to the needs of learners in a digital age. Adult participants in particular, it is claimed by Downes and Siemens, have the ability to self-manage the development of high level conceptual learning.  MOOCs are ‘demand’ driven, meeting the interests of individual students who seek out others with similar interests and the necessary expertise to support them in their learning, and for many this interest may well not include the need for deep, conceptual learning but more likely the appropriate applications of prior knowledge in new or specific contexts. MOOCs do appear to work best for those who already have a high level of education and therefore bring many of the conceptual skills developed in formal education with them when they join a MOOC, and therefore contribute to helping those who come without such skills.

Over time, as more experience is gained, MOOCs are likely to incorporate and adapt some of the findings from research on smaller group work for much larger numbers. For instance, some MOOCs are using ‘volunteer’ or community tutors (Dillenbourg, 2014).The US State Department has organized MOOC camps through US missions and consulates abroad to mentor MOOC participants. The camps include Fulbright scholars and embassy staff who lead discussions on content and topics for MOOC participants in countries abroad (Haynie, 2014). Some MOOC providers, such as the University of British Columbia, pay a small cohort of academic assistants to monitor and contribute to the MOOC discussion forums (Engle, 2014). Engle reported that the use of academic assistants, as well as limited but effective interventions from the instructors themselves, made the UBC MOOCs more interactive and engaging. However, paying for people to monitor and support MOOCs will of course increase the cost to providers. Consequently, MOOCs are likely to develop new automated ways to manage discussion effectively in very large groups. The University of Edinburgh is experimenting with automated ‘teacherbots’ that crawl through online discussion forums and direct predetermined comments to students identified as needing help or encouragement (Bayne, 2014).

These results and approaches are consistent with prior research on the importance of instructor presence for successful for-credit online learning. In the meantime, though, there is much work still to be done if MOOCs are to provide the support and structure needed to ensure deep, conceptual learning where this does not already exist in students. The development of the skills needed in a digital age is likely to be an even greater challenge when dealing with massive numbers. However, we need much more research into what participants actually learn in MOOCs and under what conditions before any firm conclusions can be drawn.

Assessment

Assessment of the massive numbers of participants in MOOCs has proved to be a major challenge. It is a complex topic that can be dealt with only briefly here. However, Chapter 5.8 provides a general analysis of different types of assessment, and Suen (2014) provides a comprehensive and balanced overview of the way assessment has been used in MOOCs to date. This section draws heavily on Suen’s paper.

Computer marked assignments

Assessment to date in MOOCs has been primarily of two kinds. The first is based on quantitative multiple-choice tests, or response boxes where formulae or ‘correct code’ can be entered and automatically checked. Usually participants are given immediate automated feedback on their answers, ranging from simple right or wrong answers to more complex responses depending on the type of response they have checked, but in all cases, the process is usually fully automated.

For straight testing of facts, principles, formulae, equations and other forms of conceptual learning where there are clear, correct answers, this works well. In fact, multiple choice computer marked assignments were used by the UK Open University as long ago as the 1970s, although the means to give immediate online feedback were not available then. However, this method of assessment is limited for testing deep or ‘transformative’ learning, and particularly weak for assessing the intellectual skills needed in a digital age, such as creative or original thinking.

Peer review

The second type of assessment that has been tried in MOOCs has been peer assessment, where participants assess each other’s work. Peer assessment is not new. It has been successfully used for formative assessment in traditional classrooms and in some online teaching for credit (Falchikov and Goldfinch, 2000; van Zundert et al., 2010). More importantly, peer assessment is seen as a powerful way to improve deep understanding and knowledge through the rating process, and at the same time, it can be useful for developing some of the skills needed in a digital age, such as critical thinking, for those participants assessing the work of others.

However, a key feature of the successful use of peer assessment has been the close involvement of an instructor or teacher, in providing benchmarks, rubrics or criteria  for assessment, and for monitoring and adjusting peer assessments to ensure consistency and a match with the benchmarks set by the instructor. Although an instructor can provide the benchmarks and rubrics in MOOCs, close monitoring of the multiple peer assessments is difficult if not impossible with the very large numbers of participants in MOOCs. As a result, MOOC participants often become incensed at being randomly assessed by other participants who may not and often do not have the knowledge or ability to give a ‘fair’ or accurate assessment of a participant’s work.

Various attempts to get round the limitations of peer assessment in MOOCs have been tried such as calibrated peer reviews, based on averaging all the peer ratings, and Bayesian post hoc stabilization, but although these statistical techniques reduce the error (or spread) of peer review somewhat they still do not remove the problems of systematic errors of judgement in raters due to misconceptions. This is particularly a problem where a majority of participants fail to understand key concepts in a MOOC, in which case peer assessment becomes the blind leading the blind.

Automated essay scoring

This is another area where there have been attempts to automate scoring. Although such methods are increasingly sophisticated they are currently limited in terms of accurate assessment to measuring primarily technical writing skills, such as grammar, spelling and sentence construction. Once again they do not measure accurately essays where higher level intellectual skills are demonstrated.

Badges and certificates

Particularly in xMOOCs, participants may be awarded a certificate or a ‘badge’ for successful completion of the MOOC, based on a final test (usually computer-marked) which measures the level of learning in a course. The American Council on Education (ACE), which represents the presidents of U.S. accredited, degree-granting institutions, recommended offering credit for five courses on the Coursera MOOC platform. However, according to the person responsible for the review process:

what the ACE accreditation does is merely accredit courses from institutions that are already accredited. The review process doesn’t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.’ (Book, 2013)

Indeed, most of the institutions offering MOOCs will not accept their own certificates for admission or credit within their own, campus-based programs. Probably nothing says more about the confidence in the quality of the assessment than this failure of MOOC providers to recognize their own teaching.

The intent behind assessment

To evaluate assessment in MOOCs requires an examination of the intent behind assessment. As identified earlier in another chapter of my book, there are many different purposes behind assessment. Peer assessment and immediate feedback on computer-marked tests can be extremely valuable for formative assessment, enabling participants to see what they have understood and to help develop further their understanding of key concepts. In cMOOCs, as Suen points out, learning is measured as the communication that takes place between MOOC participants, resulting in crowdsourced validation of knowledge – it’s what the sum of all the participants come to believe to be true as a result of participating in the MOOC, so formal assessment is unnecessary. However, what is learned in this way is not necessarily academically validated knowledge, which to be fair, is not the concern of cMOOC proponents such as Stephen Downes.

Academic assessment is a form of currency, related not only to measuring student achievement but also affecting student mobility (e.g. entrance to grad school) and perhaps more importantly employment opportunities and promotion. From a learner’s perspective, the validity of the currency – the recognition and transferability of the qualification – is essential. To date, MOOCs have been unable to demonstrate that they are able to assess accurately the learning achievements of participants beyond comprehension and knowledge of ideas, principles and processes (recognizing that there is some value in this alone). What MOOCs have not been able to demonstrate is that they can either develop or assess deep understanding or the intellectual skills required in a digital age. Indeed, this may not be possible within the constraints of massiveness, which is their major distinguishing feature from other forms of online learning, although the lack of valid methods of assessment will not stop computer scientists from trying to find ways to analyze participant online behaviour to show that such learning is taking place.

Up next

I hope the next post will be my last on this chapter on MOOCs. It will cover the following topics:

  • the cost of MOOCs and economies of scale
  • branding
  • the political, economic and social factors that explain the rise of MOOCs.

Over to you

As regular readers know, this is my way of obtaining peer review for my open textbook (so clearly I am not against peer review in principle!). So if I have missed anything important on this topic, or have misrepresented people’s views, or you just plain disagree with what I’ve written, please let me know. In particular, I am hoping for comments on:

  • comprehensiveness of the sources used that address learning and assessment methods in MOOCs
  • arguments that should have been included, either as a strength or a weakness
  • errors of fact

Yes, I’m a glutton for punishment, but you need to be a masochist to publish openly on this topic.

References

Bayne, S. (2014) Teaching, Research and the More-than-Human in Digital Education Oxford UK: EDEN Research Workshop (url to come)

Book, P. (2103) ACE as Academic Credit Reviewer–Adjustment, Accommodation, and Acceptance WCET Learn, July 25

Colvin, K. et al. (2014) Learning an Introductory Physics MOOC: All Cohorts Learn Equally, Including On-Campus Class, IRRODL, Vol. 15, No. 4

Dillenbourg, P. (ed.) (1999) Collaborative-learning: Cognitive and Computational Approaches. Oxford: Elsevier

Dillenbourg, P. (2014) MOOCs: Two Years Later, Oxford UK: EDEN Research Workshop (url to come)

Engle, W. (2104) UBC MOOC Pilot: Design and Delivery Vancouver BC: University of British Columbia

Falchikov, N. and Goldfinch, J. (2000) Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks Review of Educational Research, Vol. 70, No. 3

Firmin, R. et al. (2014) Case study: using MOOCs for conventional college coursework Distance Education, Vol. 35, No. 2

Haynie, D. (2014). State Department hosts ‘MOOC Camp’ for online learners. US News,January 20

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Ho, A. et al. (2014) HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013 (HarvardX and MITx Working Paper No. 1), January 21

Knox, J. (2014) Digital culture clash: ‘massive’ education in the e-Learning and Digital Cultures Distance Education, Vol. 35, No. 2

Kop, R. (2011) The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course International Review of Research into Open and Distance Learning, Vol. 12, No. 3

Lave, J. and Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press

Milligan, C., Littlejohn, A. and Margaryan, A. (2013) Patterns of engagement in connectivist MOOCs, Merlot Journal of Online Learning and Teaching, Vol. 9, No. 2

Suen, H. (2104) Peer assessment for massive open online courses (MOOCs) International Review of Research into Open and Distance Learning, Vol. 15, No. 3

van Zundert, M., Sluijsmans, D., van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20, 270-279

Review of ‘Online Distance Education: Towards a Research Agenda.’

Listen with webReader
Drop-out: the elephant in the DE room that no-one wants to talk about

Drop-out: the elephant in the DE room that no-one wants to talk about

Zawacki-Richter, O. and Anderson, T. (eds.) (2014) Online Distance Education: Towards a Research Agenda Athabasca AB: AU Press, pp. 508

It is somewhat daunting to review a book of over 500 pages of research on any topic. I doubt if few other than the editors are likely to read this book from cover to cover. It is more likely to be kept on one’s bookshelf (if these still exist in a digital age) for reference whenever needed. Nevertheless, this is an important work that anyone working in online learning needs to be aware of, so I will do my best to cover it as comprehensively as I can.

Structure of the book

The book is a collection of about 20 chapters by a variety of different authors (more on the choice of authors later). Based on a Delphi study and analysis of ‘key research journals’ in the field, the editors have organized the topic into three sections, with a set of chapters on each sub-section, as follows:

1. Macro-level research: distance education systems and theories

  • access, equity and ethics
  • globalization and cross-cultural issues
  • distance teaching systems and institutions
  • theories and models
  • research methods and knowledge transfer

2. Meso-level research: management, organization and technology

  • management and organization
  • costs and benefits
  • educational technology
  • innovation and change
  • professional development and faculty support
  • learner support services
  • quality assurance

3. Micro-level: teaching and learning in distance education

  • instructional/learning design
  • interaction and communication
  • learner characteristics.

In addition, there is a very useful preface from Otto Peters, an introductory chapter by the editors where they justify their structural organization of research, and a short conclusion that calls for a systematic research agenda in online distance education research.

More importantly, perhaps, Terry Anderson and Olaf Zawacki-Richter demonstrate empirically that research in this field has been skewed towards micro-level research (about half of all publications).  Interestingly, and somewhat surprisingly given its importance, costs and benefits of online distance education is the least researched area.

What I liked

It is somewhat invidious to pick out particular chapters, because different people will have different interests from such a wide-ranging list of topics. I have tended to choose those that I found were new and/or particularly enlightening for me, but other readers’ choices will be different. However, by selecting a few excellent chapters, I hope to give some idea of the quality of the book.

1. The structuring/organization of research

Anderson and Zawacki-Richter have done an excellent job in providing a structural framework for research in this field. This will be useful both for those teaching about online and distance education but in particular for potential Ph.D. students wondering what to study. This book will provide an essential starting point.

2. Summary of the issues in each area of research

Again, the editors have done an excellent job in their introductory chapter in summarizing the content of each of the chapters that follows, and in so doing pulling out the key themes and issues within each area of research. This alone makes the book worthwhile.

3. Globalization, Culture and Online Distance Education

Charlotte (Lani) Gunawardena of the University of New Mexico has written the most comprehensive and deep analysis of this issue that I have seen, and it is an area in which I have a great deal of interest, since most of the online teaching I have done has been with students from around the world and sometimes multi-lingual.

After a general discussion of the issue of globalization and education, she reviews research in the following areas:

  • diverse educational expectations
  • learners and preferred ways of learning
  • socio-cultural environment and online interaction
  • help-seeking behaviours
  • silence
  • language learning
  • researching culture and online distance learning

This chapter should be required reading for anyone contemplating teaching online.

4. Quality assurance in Online Distance Education

I picked this chapter by Colin Latchem because he is so deeply expert in this field that he is able to make what can be a numbingly boring but immensely important topic a fun read, while at the same time ending with some critical questions about quality assurance. In particular Latchem looks at QA from the following perspectives:

  • definitions of quality
  • accreditation
  • online distance education vs campus-based teaching
  • quality standards
  • transnational online distance education
  • open educational resources
  • costs of QA
  • is online distance education yet good enough?
  • an outcomes approach to QA.

This chapter definitely showcases a master at the top of his game.

5. The elephant in the room: student drop-out

This is a wonderfully funny but ultimately serious argument between Ormond Simpson and Alan Woodley about the elephant in the distance education room that no-one wants to mention. Here they start poking the elephant with some sticks (which they note is not likely to be a career-enhancing move.) The basic argument is that institutions should and could do more to reduce drop-out/increase course completion. This chapter also stunned me with providing hard data about really low completion rates for most open university students. I couldn’t help comparing these with the high completion rates for online credit courses at dual-mode (campus-based) institutions, at least in Canada (which of course are not ‘open’ institutions in that students must have good high school qualifications.)

Woodley’s solution to reducing drop-out is quite interesting (and later well argued):

  • make it harder to get in
  • make it harder to get out

In both cases, really practical and not too costly solutions are offered that nevertheless are consistent with open access and high quality teaching.

In summary

The book contains a number of really good chapters that lay out the issues in researching online distance education.

What I disliked

I have to say that I groaned when I first saw the list of contributors. The same old, same old list of distance education experts with a heavy bias towards open universities. Sure, they are nearly all well-seasoned experts, and there’s nothing wrong with that per se (after all, I see myself as one of them.)

But where are the young researchers here, and especially the researchers in open educational resources, MOOCs, social media applications in online learning, and above all researchers from the many campus-based universities now mainstreaming online learning? There is almost nothing in the book about research into blended learning, and flipped classrooms are not even mentioned. OK, the book is about online distance learning but the barriers or distinctions are coming down with a vengeance. This book will never reach those who most need it, the many campus-based instructors now venturing for the first time into online learning in one way or another. They don’t see themselves as primarily distance educators.

And a few of the articles were more like lessons in history than an up-to-date review of research in the field. Readers of this blog will know that I strongly value the history of educational technology and distance learning. But these lessons need to be embedded in the here and now. In particular, the lessons need to be spelled out. It is not enough to know that Stanford University researchers as long ago as 1974 were researching the costs and benefits of educational broadcasting in developing countries, but what lessons does this have for some of the outrageous claims being made about MOOCs? A great deal in fact, but this needs explaining in the context of MOOCs today.

Also the book is solely focused on post-secondary university education. Where is the research on online distance education in the k-12/school sector or the two-year college/vocational sector? Maybe they are topics for other books, but this is where the real gap exists in research publications in online learning.

Lastly, although the book is reasonably priced for its size (C$40), and is available as an e-text as well as the fully printed version, what a pity it is not an open textbook that could then be up-dated and crowd-sourced over time.

Conclusion

This is essential reading for anyone who wants to take a professional, evidence-based approach to online learning (distance or otherwise). It will be particularly valuable for students wanting to do research in this area. The editors have done an incredibly good job of presenting a hugely diverse and scattered area in a clear and structured manner. Many of the chapters are gems of insight and knowledge in the field.

However, we have a huge challenge of knowledge transfer in this field. Repeatedly authors in the book lamented that many of the new entrants to online learning are woefully ignorant of the research previously done in this field. We need a better way to disseminate this research than a 500 page printed text that only those already expert in the field are likely to access. On the other hand, the book does provide a strong foundation from which to find better ways to disseminate this knowledge. Knowledge dissemination in a digital world then is where the research agenda for online learning needs to focus.

 

Comparing xMOOCs and cMOOCs: philosophy and practice

Listen with webReader
They're big: but will they survive? Image: © Wikipedia

They’re big: but will they survive? Image: © Wikipedia

The story so far

For my open textbook Teaching in a Digital Age, I am writing a chapter on different design models for teaching and learning. I have started writing the section on MOOCs, and in my previous post, ‘What is a MOOC?‘, I gave a brief history and described the key common characteristics of all MOOCs.

In this post I examine the differences in philosophy and practice between xMOOCs and cMOOCs.

Design models for MOOCs

MOOCs are a relatively new phenomenon and as a result are still evolving, particularly in terms of their design. However the early MOOC courses had relatively identifiable designs which still permeate most MOOCs. At the same time, there are two quite different philosophical positions underpinning xMOOCs and cMOOCs, so we need to look at each design model separately.

xMOOCs

I am starting with xMOOCs because at the time of writing they are by far the most common MOOC. Because instructors have considerable flexibility in the design of the course, there is considerable variation in the details, but in general xMOOCs have the following common design features:

  • specially designed platform software: xMOOCs use specially designed platform software that allows for the registration of very large numbers of participants, provides facilities for the storing and streaming on demand of digital materials, and automates assessment procedures and student performance tracking.
  • video lectures: xMOOCs use the standard lecture mode, but delivered online by participants downloading on demand recorded video lectures. These video lectures are normally available on a weekly basis over a period of 10-13 weeks. Initially these were often 50 minute lectures, but as a result of experience some xMOOCs now are using shorter recordings (sometimes down to 15 minutes in length) and thus there may be more video segments. Over time, xMOOC courses, as well as the videos, are becoming shorter in length, some now lasting only five weeks. Various video production methods have been used, including lecture capture (recording face-to-face on-campus lectures, then storing them and streaming them on demand), full studio production, or desk-top recording by the instructor on their own.
  • computer-marked assignments: students complete an online test and receive immediate computerised feedback. These tests are usually offered throughout the course, and may be used just for participant feedback. Alternatively the tests may be used for determining the award of a certificate. Another option is for an end of course grade or certificate based solely on an end-of-course online test. Most xMOOC assignments are based on multiple-choice, computer-marked questions, but some MOOCs have also used text or formula boxes for participants to enter answers, such as coding in a computer science course, or mathematical formulae, and in one or two cases, short text answers, but in all cases these are computer-marked.
  • peer assessment: some xMOOCs have experimented with assigning students randomly to small groups for peer assessment, especially for more open-ended or more evaluative assignment questions. This has often proved problematic though because of wide variations in expertise between the different members of a group, and because of the different levels of involvement in the course of different participants.
  • supporting materials: sometimes copies of slides, supplementary audio files, urls to other resources, and online articles may be included for downloading by participants.
  • a shared comment/discussion space where participants can post questions, ask for help, or comment on the content of the course.
  • no or very light discussion moderation: the extent to which the discussion or comments are moderated varies probably more than any other feature in xMOOCs, but at its most, moderation is directed at all participants rather than to individuals. Because of the very large numbers participating and commenting, moderation of individual comments by the instructor(s) offering the MOOC is impossible. Some instructors offer no moderation whatsoever, so participants rely on other participants to respond to questions or comments. Some instructors ‘sample’ comments and questions, and post comments in response to these. Some instructors use teaching assistants to comb for or identify common areas of concern shared by a number of participants then the instructor or teaching assistants will respond. However, in most cases, participants moderate each other’s comments or questions.
  • badges or certificates: most xMOOCs award some kind of recognition for successful completion of a course, based on a final computer-marked assessment. However, at the time of writing, MOOC badges or certificates have not been recognised for credit or admission purposes even by the institutions offering a MOOC, or even when the lectures are the same as for on-campus students. No evidence exists to date about employer acceptance of MOOC qualifications.
  • learning analytics: Although to date there has not been a great deal of published information about the use of learning analytics in xMOOCs, the xMOOC platforms have the capacity to collect and analyse ‘big data’ about participants and their performance, enabling, at least in theory, for immediate feedback to instructors about areas where the content or design needs improving and possibly directing automated cues or hints for individuals.

xMOOCs therefore primarily use a teaching model focused on the transmission of information, with high quality content delivery, computer-marked assessment (mainly for student feedback purposes), and automation of all key transactions between participants and the learning platform. There is almost no direct interaction between an individual participant and the instructor responsible for the course.

cMOOCs

cMOOCs have a very different educational philosophy from xMOOCs, in that cMOOCs place heavy emphasis on networking and in particular on strong content contributions from the participants themselves.

Key design principles

Downes (2014) has identified four key design principles for cMOOCs:

  • autonomy of the learner: in terms of learners choosing what content or skills they wish to learn, learning is personal, and thus there being no formal curriculum
  • diversity: in terms of the tools used, the range of participants and their knowledge levels, and varied content
  • interactivity: in terms of co-operative learning, communication between participants, resulting in emergent knowledge
  • open-ness: in terms of access, content, activities and assessment

Thus for the proponents of cMOOCs, learning results not from the transmission of information from an expert to novices, as in xMOOCs, but from sharing of knowledge between participants.

From principles to practice

Identifying how these key design features for cMOOCs are turned into practice is somewhat more difficult to pinpoint, because cMOOCs depend on an evolving set of practices. Most cMOOCs to date have in fact made some use of ‘experts’, both in the organization and promotion of the MOOC, and in providing ‘nodes’ of content around which discussion tends to revolve.  In other words, the design practices of cMOOCs are still more a work in progress than those of xMOOCs.

Nevertheless, I see the following as key design practices to date in cMOOCs:

  • use of social media: partly because most cMOOCs are not institutionally based or supported, they do not at present use a shared platform or platforms but are more loosely supported by a range of ‘connected’ tools and media. These may include a simple online registration system, and the use of web conferencing tools such as Blackboard Collaborate or Adobe Connect, streamed video or audio files, blogs, wikis, ‘open’ learning management systems such as Moodle or Canvas, Twitter, LinkedIn or Facebook, all enabling participants to share their contributions. Indeed, as new apps and social media tools develop, they too are likely to be incorporated into cMOOCs. All these tools are connected through web-based hashtags or other web-based linking mechanisms, enabling participants to identify social media contributions from other participants. Downes (2014) is working on a Learning and Performance Support System that could be used to help both participants and cMOOC organisers to communicate more easily across the whole MOOC and to organise their personal learning. Thus the use of loosely linked/connected social media is a key design practice in cMOOCs
  • participant-driven content: in principle, other than a common topic that may be decided by someone wanting to organise a cMOOC, content is decided upon and contributed by the participants themselves, in this sense very much like any other community of practice. In practice though cMOOC organisers (who themselves tend to have some expertise in the topic of the cMOOC) are likely to invite potential participants who have expertise or are known already to have a well articulated approach to a topic to make contributions around which participants can discuss and debate. Other participants choose their own ways to contribute or communicate, the most common being through blog posts, tweets, or comments on other participants’ blog posts, although some cMOOCs use wikis or open source online discussion forums. The key design practice with regard to content is that all participants contribute to and share content.
  • distributed communication: this is probably the most difficult design practice to understand for those not familiar with cMOOCs – and even for those who have participated. With participants numbering in the hundreds or even thousands, each contributing individually through a variety of social media, there are a myriad different inter-connections between participants that are impossible to track (in total) for any single participant. This results in many sub-conversations, more commonly at a binary level of two people communicating with each other than an integrated group discussion, although all conversations are ‘open’ and all other participants are able to contribute to a conversation if they know it exists. The key design practice then with regard to communication is a self-organising network with many sub-components.
  • assessment: there is no formal assessment, although participants may seek feedback from other, more knowledgeable participants, on an informal basis. Basically participants decide for themselves whether what they have learned is appropriate to them.

cMOOCs therefore primarily use a networked approach to learning based on autonomous learners connecting with each other across open and connected social media and sharing knowledge through their own personal contributions. There is no pre-set curriculum and no formal teacher-student relationship, either for delivery of content or for learner support. Participants learn from the contributions of others, from the meta-level knowledge generated through the community, and from self-reflection on their own contributions.

This is very much a personal interpretation of how cMOOCs work in practice, based largely on my own experience as a participant, but much more has been written and spoken about the philosophy of cMOOCs, and much less about the implementation of that philosophy, presumably because cMOOC proponents want to leave it open to practitioners to decide how best to put that philosophy into practice.

What is clear though is that Downes was correct in clearly distinguishing cMOOCs from xMOOCs – they are very different beasts.

Coming next to a web page near you

Now for the fun part. Over the next few days I will be writing about the strengths and weaknesses of MOOCs, focusing particularly on the following question:

Can or do MOOCs provide the learning and skills that students will need in the future? 

I can in fact provide you with the short answer now: a resounding NO, for both kinds of MOOC, although one is a bit better than the other! Tune in later for the full details.

Feedback, please

In the meantime, I need to know whether I have got it right in describing the two kinds of MOOCs. Does my description – because that is all it’s meant to be at this stage – match your experience of MOOCs? Have I missed important characteristics? Do I have my facts wrong? Is this useful or is there a better way to approach this topic?