December 16, 2017

Automation or empowerment: online learning at the crossroads

Image: Applift

Image: AppLift, 2015

You are probably, like me, getting tired of the different predictions for 2016. So I’m not going to do my usual look forward for the year for individual developments in online learning. Instead, I want to raise a fundamental question about which direction online learning should be heading in the future, because the next year could turn out to be very significant in determining the future of online learning.

The key question we face is whether online learning should aim to replace teachers and instructors through automation, or whether technology should be used to empower not only teachers but also learners. Of course, the answer will always be a mix of both, but getting the balance right is critical.

An old but increasingly important question

This question, automation or human empowerment, is not new. It was raised by B.F. Skinner (1968) when he developed teaching machines in the early 1960s. He thought teaching machines would eventually replace teachers. On the other hand, Seymour Papert (1980) wanted computing to empower learners, not to teach them directly. In the early 1980s Papert got children to write computer code to improve the way they think and to solve problems. Papert was strongly influenced by Jean Piaget’s theory of cognitive development, and in particular that children constructed rather than absorbed knowledge.

In the 1980s, as personal computers became more common, computer-assisted learning (CAL or CAD) became popular, using computer-marked tests and early forms of adaptive learning. Also in the 1980s the first developments in artificial intelligence were applied, in the form of intelligent math tutoring. Great predictions were made then, as now, about the potential of AI to replace teachers.

Then along came the Internet. Following my first introduction to the Internet in a friend’s basement in Vancouver, I published an article in the first edition of the Journal of Distance Education, entitled ‘Computer-assisted learning or communications: which way for IT in distance education?’ (1986). In this paper I argued that the real value of the Internet and computing was to enable asynchronous interaction and communication between teacher and learners, and between learners themselves, rather than as teaching machines. This push towards a more constructivist approach to the use of computing in education was encapsulated in Mason and Kaye’s book, Mindweave (1989). Linda Harasim has since argued that online collaborative learning is an important theory of learning in its own right (Harasim, 2012).

In the 1990s, David Noble of York University attacked online learning in particular for turning universities into ‘Digital Diploma Mills’:

‘universities are not only undergoing a technological transformation. Beneath that change, and camouflaged by it, lies another: the commercialization of higher education.’

Noble (1998) argued that

‘high technology, at these universities, is often used not to ……improve teaching and research, but to replace the visions and voices of less-prestigious faculty with the second-hand and reified product of academic “superstars”.

However, contrary to Noble’s warnings, for fifteen years most university online courses followed more the route of interaction and communication between teachers and students than computer-assisted learning or video lectures, and Noble’s arguments were easily dismissed or forgotten.

Then along came lecture capture and with it, in 2011, Massive Open Online Courses (xMOOCs) from Coursera, Udacity and edX, driven by elite, highly selective universities, with their claims of making the best professors in the world available to everyone for free. Noble’s nightmare suddenly became very real. At the same time, these MOOCs have resulted in much more interest in big data, learning analytics, a revival of adaptive learning, and claims that artificial intelligence will revolutionize education, since automation is essential for managing such massive courses.

Thus we are now seeing a big swing back to the automation of learning, driven by powerful computing developments, Silicon Valley start-up thinking, and a sustained political push from those that want to commercialize education (more on this later). Underlying these developments is a fundamental conflict of philosophies and pedagogies, with automation being driven by an objectivist/behaviourist view of the world, compared with the constructivist approaches of online collaborative learning.

In other words, there are increasingly stark choices to be made about the future of online learning. Indeed, it is almost too late – I fear the forces of automation are winning – which is why 2016 will be such a pivotal year in this debate.

Automation and the commercialization of education

These developments in technology are being accompanied by a big push in the United States, China, India and other countries towards the commercialization of online learning. In other words, education is being seen increasingly as a commodity that can be bought and sold. This is not through the previous and largely discredited digital diploma mills of the for-profit online universities such as the University of Phoenix that David Noble feared, but rather through the encouragement and support of commercial computer companies moving into the education field, companies such as Coursera, Lynda.com and Udacity.

Audrey Watters and EdSurge both produced lists of EdTech ‘deals’ in 2015 totalling between $1-$2 billion. Yes, that’s right, that’s $1-$2 billion in investment in private ed tech companies in the USA (and China) in one year alone. At the same time, entrepreneurs are struggling to develop sustainable business models for ed tech investment, because with education funded publicly, a ‘true’ market is restricted. Politicians, entrepreneurs and policy makers on the right in the USA increasingly see a move to automation as a way of reducing government expenditure on education, and one means by which to ‘free up the market’.

Another development that threatens the public education model is the move by very rich entrepreneurs such as the Gates, the Hewletts and the Zuckerbergs to move their massive personal wealth into ‘charitable’ foundations or corporations and use this money for their pet ‘educational’ initiatives that also have indirect benefits for their businesses. Ian McGugan (2015) in the Globe and Mail newspaper estimates that the Chan Zuckerberg Initiative is worth potentially $45 billion, and one of its purposes is to promote the personalization of learning (another name hi-jacked by computer scientists; it’s a more human way of describing adaptive learning). Since one way Facebook makes its money is by selling personal data, forgive my suspicions that the Zuckerberg initiative is a not-so-obvious way of collecting data on future high earners. At the same time, the Chang Zuckerberg initiative enables the Zuckerberg’s to avoid paying tax on their profits from Facebook. Instead then of paying taxes that could be used to support public education, these immensely rich foundations enable a few entrepreneurs to set the agenda for how computing will be used in education.

Why not?

Technology is disrupting nearly every other business and profession, so why not education? Higher education in particular requires a huge amount of money, mostly raised through taxes and tuition fees, and it is difficult to tie results directly to investment. Surely we should be looking at ways in which technology can change higher education so that it is more accessible, more affordable and more effective in developing the knowledge and skills required in today’s and tomorrow’s society?

Absolutely. It is not so much the need for change that I am challenging, but the means by which this change is being promoted. In essence, a move to automated learning, while saving costs, will not improve the learning that matters, and particularly the outcomes needed in a digital age, namely, the high level intellectual skills of critical thinking, innovation, entrepreneurship, problem-solving , high-level multimedia communication, and above all, effective knowledge management.

To understand why automated approaches to learning are inappropriate to the needs of the 21st century we need to look particularly at the tools and methods being proposed.

The problems with automating learning

The main challenge for computer-directed learning such as information transmission and management through Internet-distributed video lectures, computer-marked assessments, adaptive learning, learning analytics, and artificial intelligence is that they are based on a model of learning that has limited applications. Behaviourism works well in assisting rote memory and basic levels of comprehension, but does not enable or facilitate deep learning, critical thinking and the other skills that are essential for learners in a digital age.

R. and D. Susskind (2015) in particular argue that there is a new age in artificial intelligence and adaptive learning driven primarily by what they call the brute force of more powerful computing. Why AI failed so dramatically in the 1980s, they argue, was because computer scientists tried to mimic the way that humans think, and computers then did not have the capacity to handle information in the way they do now. When however we use the power of today’s computing, it can solve previously intractable problems through analysis of massive amounts of data in ways that humans had not considered.

There are several problems with this argument. The first is that the Susskinds are correct in that computers operate differently from humans. Computers are mechanical and work basically on a binary operating system. Humans are biological and operate in a far more sophisticated way, capable of language creation as well as language interpretation, and use intuition as well as deductive thinking. Emotion as well as memory drives human behaviour, including learning. Furthermore humans are social animals, and depend heavily on social contact with other humans for learning. In essence humans learn differently from the way machine automation operates.

Unfortunately, computer scientists frequently ignore or are unaware of the research into human learning. In particular they are unaware that learning is largely developmental and constructed, and instead impose an old and less appropriate method of teaching based on behaviourism and an objectivist epistemology. If though we want to develop the skills and knowledge needed in a digital age, we need a more constructivist approach to learning.

Supporters of automation also make another mistake in over-estimating or misunderstanding how AI and learning analytics operate in education. These tools reflect a highly objectivist approach to teaching, where procedures can be analysed and systematised in advance. However, although we know a great deal about learning in general, we still know very little about how thinking and decision-making operate biologically in individual cases. At the same time, although brain research is promising to unlock some of these secrets, most brain scientists argue that while we are beginning to understand the relationship between brain activity and very specific forms of behaviour, there is a huge distance to travel before we can explain how these mechanisms affect learning in general or how an individual learns in particular. There are too many variables (such as emotion, memory, perception, communication, as well as neural activity) at play to find an isomorphic fit between the firing of neurons and computer ‘intelligence’.

The danger then with automation is that we drive humans to learn in ways that best suit how machines operate, and thus deny humans the potential of developing the higher levels of thinking that make humans different from machines. For instance, humans are better than machines at dealing with volatile, uncertain, complex and ambiguous situations, which is where we find ourselves in today’s society.

Lastly, both AI and adaptive learning depend on algorithms that predict or direct human behaviour. These algorithms though are not transparent to the end users. To give an example, learning analytics are being used to identify students at high risk of failure, based on correlations of previous behaviour online by previous students. However, for an individual, should a software program be making the decision as to whether that person is suitable for higher education or a particular course? If so, should that person know the grounds on which they are considered unsuitable and be able to challenge the algorithm or at least the principles on which that algorithm is based? Who makes the decision about these algorithms – a computer scientist using correlated data, or an educator concerned with equitable access? The more we try to automate learning, the greater the danger of unintended consequences, and the more need for educators rather than computer scientists to control the decision-making.

The way forward

In the past, I used to think of computer scientists as colleagues and friends in designing and delivering online learning. I am now increasingly seeing at least some of them as the enemy. This is largely to do with the hubris of Silicon Valley, which believes that computer scientists can solve any problem without knowing anything about the problem itself. MOOCs based on recorded lectures are a perfect example of this, being developed primarily by a few computer scientists from Stanford (and unfortunately blindly copied by many people in universities who should have known better.)

We need to start with the problem, which is how do we prepare learners for the knowledge and skills they will need in today’s society. I have argued (Bates, 2015) that we need to develop, in very large numbers of people, high level intellectual and practical skills that require the construction and development of knowledge, and that enable learners to find, analyse, evaluate and apply knowledge appropriately.

This requires a constructivist approach to learning which cannot be appropriately automated, as it depends on high quality interaction between knowledge experts and learners. There are many ways to accomplish this, and technology can play a leading role, by enabling easy access to knowledge, providing opportunities for practice in experientially-based learning environments, linking communities of scholars and learners together, providing open access to unlimited learning resources, and above all by enabling students to use technology to access, organise and demonstrate their knowledge appropriately.

These activities and approaches do not easily lend themselves to massive economies of scale through automation, although they do enable more effective outcomes and possibly some smaller economies of scale. Automation can be helpful in developing some of the foundations of learning, such as basic comprehension or language acquisition. But at the heart of developing the knowledge and skills needed in today’s society, the role of a human teacher, instructor or guide will remain absolutely essential. Certainly, the roles of teachers and instructors will need to change quite dramatically, teacher training and faculty development will be critical for success, and we need to use technology to enable students to take more responsibility for their own learning, but it is a dangerous illusion to believe that automation is the solution to learning in the 21st century.

Protecting the future

There are several practical steps that need to be taken to prevent the automation of teaching.

  1. Educators – and in particular university presidents and senior civil servants with responsibility for education – need to speak out clearly about the dangers of automation, and the technology alternatives available that still exploit its potential and will lead to greater cost-effectiveness. This is not an argument against the use of technology in education, but the need to use it wisely so we get the kind of educated population we need in the 21st century.
  2. Computer scientists need to show more respect to educators and be less arrogant. This means working collaboratively with educators, and treating them as equals.
  3. We – teachers and educational technologists – need to apply in our own work and disseminate better to those outside education what we already know about effective learning and teaching.
  4. Faculty and teachers need to develop compelling technology alternatives to automation that focus on the skills and knowledge needed in a digital age, such as:
    • experiential learning through virtual reality (e.g. Loyalist College’s training of border service agents)
    • networking learners online with working professionals, to solve real world problems (e.g. by developing a program similar to McMaster’s integrated science program for online/blended delivery)
    • building strong communities of practice through connectivist MOOCs (e.g. on climate change or mental health) to solve global problems
    • empowering students to use social media to research and demonstrate their knowledge through multimedia e-portfolios (e.g. UBC’s ETEC 522)
    • designing openly accessible high quality, student-activated simulations and games but designed and monitored by experts in the subject area.
  5. Governments need to put as much money into research into learning and educational technology as they do into innovation in industry. Without better and more defensible theories of learning suitable for a digital age, we are open to any quack or opportunist who believes he or she has the best snake oil. More importantly, with better theory and knowledge of learning disseminated and applied appropriately, we can have a much more competitive workforce and a more just society.
  6. We need to educate our politicians about the dangers of commercialization in education through the automation of learning and fight for a more equal society where the financial returns on technology applications are more equally shared.
  7. Become edupunks and take back the web from powerful commercial interests by using open source, low cost, easy to use tools in education that protect our privacy and enable learners and teachers to control how they are used.

That should keep you busy in 2016.

Your views are of course welcome – unless you are a bot.

References

Bates, A. (1986) Computer assisted learning or communications: which way for information technology in distance education? Journal of Distance Education Vol. 1, No. 1

Bates, A. (2015) Teaching in a Digital Age Victoria BC: BCcampus

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Mason, R. and Kaye, A (Eds).(1989)  Mindweave: communication, computers and distance education. Oxford: Pergamon

McGugan, I. (2015)Why the Zuckerberg donation is not a bundle of joy, Globe and Mail, December 2

Noble, D. (1998) Digital Diploma Mills, Monthly Review http://monthlyreview.org/product/digital_diploma_mills/

Papert, S. (1980) Mindstorms: Children, Computers and Powerful Ideas New York: Basic Books

Skinner, B. (1968)  The Technology of Teaching, 1968 New York: Appleton-Century-Crofts

Susskind, R. and Susskind, D. (2015) The Future of the Professions: How Technology will Change the Work of Human Experts Oxford UK: Oxford University Press

Watters, A. (2015) The Business of EdTech, Hack Edu, undated http://2015trends.hackeducation.com/business.html

Winters, M. (2015) Christmas Bonus! US Edtech Sets Record With $1.85 Billion Raised in 2015 EdSurge, December 21 https://www.edsurge.com/news/2015-12-21-christmas-bonus-us-edtech-sets-record-with-1-85-billion-raised-in-2015

Last chapter of Teaching in a Digital Age now published

Books lots! 2

Chapter 12, Supporting teachers and instructors in a digital age, the last chapter of my online, open textbook for teachers and instructors, Teaching in a Digital Age, is now published. It covers the following:

 

 

Section 12.7 is really a summary of the main points in the book, which I reproduce below as the key takeaways from the book.

I will do a separate post on Scenario G, which provides a possible future scenario for teaching in a digital age.

The book is by no means finished. I need to do some serious editing, but the book now exists in a form that can be used immediately for supporting faculty development, or for teachers and instructors interested in improving their teaching.

 

Key Takeaways

1. There is increasing pressure from employers, the business community, learners themselves, and also from  a significant number of educators, for learners to develop the type of knowledge and the kinds of skills that they will need in a digital age.

2. The knowledge and skills needed in a digital age, where all ‘content’ will be increasingly and freely available over the Internet, requires graduates with expertise in:

  • knowledge management (the ability to find, evaluate and appropriately apply knowledge),
  • IT knowledge and skills,
  • inter-personal communication skills, including the appropriate use of social media
  • independent and lifelong learning skills
  • a range of intellectual skills, including
    • knowledge construction
    • reasoning
    • critical analysis,
    • problem-solving,
    • creativity
  • collaborative learning and teamwork
  • multi-tasking and flexibility.

These are all skills that are relevant to any subject domain, and need to be embedded within that domain. With such skills, graduates will be better prepared for a volatile, uncertain, complex and ambiguous world.

3. To develop such knowledge and skills, teachers and instructors need to set clear learning outcomes and select teaching methods  that will support the development of such knowledge and skills, and, since all skills require practice and feedback to develop, learners must be given ample opportunity to practice such skills. This requires moving away from a model of information transmission to greater student engagement, more learner-centred teaching, and new methods of assessment that measure skills as well as mastery of content.

4. Because of the increased diversity of students, from full-time campus-based learners to lifelong learners already with high levels of post-secondary education to learners who have slipped through the formal school system and need second-chance opportunities, and because of the capacity of new information technologies to provide learning at any time and any place, a much wider range of modes of delivery are needed, such as campus-based teaching, blended or hybrid learning and fully online courses and programs, both in formal and in non-formal settings.

5. The move to blended, hybrid and online learning and a greater use of learning technologies offers more options and choices for teachers and instructors. In order to use these technologies well, teachers and instructors require not only to know the strengths and weaknesses of different kinds of technology, but also need to have a good grasp of how students learn best. This requires knowing about

  • the research into teaching and learning,
  • different theories of learning related to different concepts of knowledge (epistemology),
  • different methods of teaching and their strengths and weaknesses.

Without this basic foundation, it is difficult for teachers and instructors to move away from the only model that many are familiar with, namely the lecture and discussion model, which is limited in terms of developing the knowledge and skills required in a digital age.

6. The challenge is particularly acute in universities. There is no requirement to have any training or qualification in teaching to work in a university in most Western countries. Nevertheless teaching will take up a minimum of 40 per cent of a faculty member’s time, and much more for many adjunct or contract faculty or full time college instructors. However, the same challenge remains, to a lesser degree, for school teachers and college instructors: how to ensure that already experienced professionals have the knowledge and skills required to teach well in a digital age.

7. Institutions can do much to facilitate or impede the development of the knowledge and skills required in a digital age. They need to

  • ensure that all levels of teaching and instructional staff have adequate training in the new technologies and methods of teaching necessary for the development of the knowledge and skills required in a digital age
  • ensure that there is adequate learning technology support for teachers and instructors
  • ensure that conditions of employment and in particular class size enable teaching and instructional staff to teach in the ways that will develop the knowledge and skills needed in a digital age
  • develop a practical and coherent institutional strategy to support he kind of teaching needed in a digital age.

8. Although governments, institutions and learners themselves can do a great deal to ensure success in teaching and learning, in the end the responsibility and to some extent the power to change lies within teachers and instructors themselves.

9. It will be the imagination of teachers inventing new ways of teaching that will eventually result in the kinds of graduates the world will need in the future

 Your turn

I’m now in the final editing stages. The book will be available for review and I will be approaching some of the leading experts in this area to do a full critique and suggestions for improvement. But now is your chance. If you have:

  • criticisms of what I’ve written
  • suggestions for adding things that I missed
  • suggestions for improvements to content
  • suggestions for improvements to the open textbook format
  • any other comments, negative or positive

about the whole book, please let me know.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Seeking the unique pedagogical characteristics of text and print

There's nothing like a good book - or is there?

There’s nothing like a good book – or is there?

This is the first of several posts on the unique characteristics of different media, for my open textbook, Teaching in a Digital Age. I’m starting with text, because it is – or perhaps more accurately, has been – fundamental to the development of academic knowledge. However, writing about its unique pedagogical features is rather like asking a fish to describe water. We are so immersed in text in academia that it is hard to imagine studying without texts to read and learn from.

However, with the increasing availability of other media, what is so special about text? How does it differ from other media? I have found writing about this particularly difficult. I have lots of empirical evidence on the pedagogical influences of audio, video and computing, but almost nothing on text, because in a sense it is the default medium for academic learning, the base against which other media tend to be judged. Now much has been published on what makes for good writing, and even what makes for good academic writing, but that is different from asking what can text do for learning that is unique from other media.

As a result, the following section strikes me as being rather unacademic, more of an opinion piece than an empirically supported and theoretically based account of the strengths and weaknesses of text as a teaching medium. So please bear this in mind when reading it, and if you have suggestions for improving it, or other work of which I should be aware, please provide feedback.

The unique pedagogical features of text

Ever since the invention of the Gutenberg press, print has been a dominant teaching technology, arguably at least as influential as the spoken word of the teacher. Even today, textbooks, mainly in printed format, but increasingly also in digital format, still play a major role in formal education, training and distance education. Many fully online courses still make extensive use of text-based learning management systems and online asynchronous discussion forums.

Why is this? What makes text such a powerful teaching medium, and will it remain so, given the latest developments in information technology?

In essence, I am arguing that the unique pedagogical characteristics of text are as follows:

  • text is particularly good at handling abstraction and generalisation, mainly through written language
  • text enables the linear sequencing of information in a structured format
  • text can present and separate empirical evidence or data from the abstractions, conclusions or generalisations derived from the empirical evidence
  • text’s linear structure enables the development of coherent, sequential argument or discussion
  • at the same time text can relate evidence to argument and vice versa
  • text’s recorded and permanent nature enables independent analysis and critique of its content

There is some overlap of each of these features with other media, but no other medium combines all these characteristics, or is as powerful as text with respect to these characteristics.

Text can come in many formats, including printed textbooks, text messages, novels, magazines, newspapers, scribbled notes, journal articles, essays, novels, online asynchronous discussions and so on. I want to focus particularly on the role of the book, because of its centrality in academic learning.

The book and knowledge

Earlier (Chapter 2, Section 2.4,) I argued that academic knowledge is a specific form of knowledge that has characteristics that differentiate it from other kinds of knowledge, and particularly from knowledge or beliefs based solely on direct personal experience. Academic knowledge is a second-order form of knowledge that seeks abstractions and generalizations based on reasoning and evidence.

Fundamental components of academic knowledge are:

  • codification: knowledge can be consistently represented in some form (words, symbols, video)
  • transparency: the source of the knowledge can be traced and verified
  • reproduction: knowledge can be reproduced or have multiple copies
  • communicability: knowledge must be in a form such that it can be communicated and challenged by others.

The book has proved to be a remarkably powerful medium for the development and transmission of academic knowledge, since it meets all four criteria above, but to what extent can new media such as blogs, wikis, multimedia, and social media replace the book in academic knowledge? New media can in fact handle just as well some of these criteria, and provide indeed added value, such as speed of reproduction and ubiquity, but the book still has some unique qualities. A key advantage of a book is that it allows for the development of a sustained, coherent, and comprehensive argument with evidence to support the argument. Blogs can do this only to a limited extent (otherwise they cease to be blogs and become articles or a digital book).

Quantity is important sometimes and books allow for the collection of a great deal of evidence and supporting argument, and allow for a wider exploration of an issue or theme, within a relatively condensed and portable format. A consistent and well supported argument, with evidence, alternative explanations or even counter positions, requires the extra ‘space’ of a book. Above all, books can provide coherence or a sustained, particular position or approach to a problem or issue, a necessary balance to the chaos and confusion of the many new forms of digital media that constantly compete for our attention, but in much smaller ‘chunks’ that are overall more difficult to integrate and digest.

Another important academic feature of text is that it can be carefully scrutinised, analysed and constantly checked, partly because it is largely linear, and also permanent once published, enabling more rigorous challenge or testing in terms of evidence, rationality, and consistency. Multimedia in recorded format can come close to meeting these criteria, but text can also provide more convenience and in media terms, more simplicity. For instance I repeatedly find analysing video, which incorporates many variables and symbol systems, more complex than analysing a linear text, even if both contain equally rigorous (or equally sloppy) arguments.

Form and function

Does the form or technological representation of a book matter any more? Is a book still a book if downloaded and read on an iPad or Kindle, rather than as printed text?

For the purposes of knowledge acquisition, it probably isn’t any different. Indeed, for study purposes, a digital version is probably more convenient because carrying an iPad around with maybe hundreds of books downloaded on it is certainly preferable to carrying around the printed versions of the same books. There are still complaints by students about the difficulties of annotating e-books, but this will almost certainly become a standard feature available for e-books in the future.

If the whole book is downloaded, then the function of a book doesn’t change much just because it is available digitally. However, there are some subtle changes. Some would argue that scanning is still easier with a printed version. Have you ever had the difficulty of finding a particular quotation in a digital book compared with the printed version? Sure, you can use the search facility, but that means knowing exactly the correct words or the name of the person being quoted. With a printed book, I can often find a quotation just by flicking the pages, because I am using context and rapid eye scanning to locate the source, even when I don’t know exactly what I am looking for. On the other hand, searching when you do know what you are looking for (e.g. a reference by a particular author) is much easier digitally.

The other thing that happens when books are digitally available is that often, users can download only the selected chapters that are of interest to them. This is valuable if you know just what you want, but there are also dangers. For instance in my book on the strategic management of technology, the last chapter summarizes the rest of the book. If the book had been digital, the temptation then would be to just download the final chapter. You’d have all the important messages in the book, right? Well, no. What you would be missing is the evidence for the conclusions. Now the book on strategic management is based on case studies, so it would be really important to check back with how the case studies were interpreted to get to the conclusions, as this will affect the confidence you would have as a reader in the conclusions that were drawn. If just the digital version of only the last chapter is downloaded, you also lose the context of the whole book. Having the whole book gives readers more freedom to interpret and add their own conclusions than just having a summary chapter.

In conclusion, then, there are advantages and disadvantages of digitizing a book, but the essence of a book is not greatly changed when it becomes digital rather than printed.

A new niche for books in academia

We have seen historically that new media often do not entirely replace an older medium, but the old medium finds a new ‘niche’. Thus television did not lead to the complete demise of radio. Similarly, I suspect that there will be a continued role for the book in academic knowledge, enabling the book (whether digital or printed) to thrive alongside new media and formats in academia.

However, books that retain their value academically will likely need to be much more specific in their format and their purpose than has been the case to date. For instance, I see the end of books consisting mainly of a collection of loosely connected but semi-independent chapters from different authors, unless there is a strong cohesion and edited presence that provides an integrated argument or consistent set of data across all the chapters. Most of all, books may need to change some of their features, to allow for more interaction and input from readers, and more links to the outside world. It is much more unlikely though that books will survive in a printed format, because digital publication allows for many more features to be added, reduces the environmental footprint, and is much more portable and transferable.

Lastly, this is not an argument for ignoring the academic benefits of new media. The value of graphics, video and animation for representing knowledge, the ability to interact asynchronously with other learners, and the value of social networks, are all under-exploited in academia. But text and books are still important.

For another perspective on this, see Clive Shepherd’s blog: Weighing up the benefits of traditional book publishing

Text and other forms of knowledge

I have focused particularly on text and academic knowledge, because of the traditional importance of text and printed knowledge in academia. The unique pedagogical characteristics of text though may be less for other forms of knowledge. Indeed, multimedia may have many more advantages in vocational and technical education, as we shall see.

In the k-12 or school sector, text and print are likely to remain important, because reading and writing are likely to remain important – perhaps even more important – in a digital age, so the study of text (digital and printed) will remain important if only for developing literacy skills.

More evidence, please

Although there has been extensive research on the pedagogical features of other media such as audio, video and computing, text has generally been treated as the default mode, the base against which other media are compared. As a result print in particular is largely taken for granted in academia. We are now though at the stage where we need to pay much more attention to the unique characteristics of text in its various formats, in relation to other media. Until though we have more empirical studies on the unique characteristics of text and print, it would be unwise to reject the value of text for academic learning.

Feedback

I am so unsure about this section I am tempted to publish it as ‘Still under construction.’ Ideally, I’d like to link this section to a better source, as I feel it is so inadequate. So if you are in a position to offer any help or suggestions, I will be extremely grateful, as will readers of the book.

Up next

The unique pedagogical characteristics of audio.

Two design models for online collaborative learning: same or different?

Image: © Campaign Brief, 2013

Image: © Campaign Brief, 2013

I am now about to wrap up my Chapter 5 on Design models, for my open textbook, Teaching in a Digital Age.

Here I am looking at the work of two separate and important Canadian theorists and practitioners, what we might call the Toronto school, Linda Harasim and her former colleagues at the Ontario Institute for Studies in Education (OISE) in Toronto (although Linda has been firmly based for 25 years at SFU in Vancouver/Burnaby), and the Alberta school, Randy Garrison, and colleagues Terry Anderson and Walter Archer. However, they are not the only contributors to the design of online collaborative learning, as the following post makes clear.

Perhaps more importantly, I believe that online collaborative learning is a key model for teaching the knowledge and skills needed in a digital age. So here’s my first draft:

From the quite early days of online learning, some instructors have focused heavily on the communication affordances of the Internet. They have based their teaching on the concept of knowledge construction, the gradual building of knowledge mainly through asynchronous online discussion among students and between students and an instructor.

What is online collaborative learning?

The concurrence of both constructivist approaches to learning and the development of the Internet has led to the development of a particular form of constructivist teaching, originally called computer-mediated communication (CMC), or networked learning, but which has been developed into what Harasim (2012) now calls online collaborative learning theory (OCL). She describes OCL as follows (p. 90):

OCL theory provides a model of learning in which students are encouraged and supported to work together to create knowledge: to invent, to explore ways to innovate, and, by so doing, to seek the conceptual knowledge needed to solve problems rather than recite what they think is the right answer. While OCL theory does encourage the learner to be active and engaged, this is not considered to be sufficient for learning or knowledge construction……In the OCL theory, the teacher plays a key role not as a fellow-learner, but as the link to the knowledge community, or state of the art in that discipline. Learning is defined as conceptual change and is key to building knowledge. Learning activity needs to be informed and guided by the norms of the discipline and a discourse process that emphasises conceptual learning and builds knowledge.

OCL builds on and integrates theories of cognitive development that focus on conversational learning (Pask, 1975), conditions for deep learning (Marton and Saljø, 1997; Entwistle, 2000), development of academic knowledge (Laurillard, 2000) and knowledge construction (Scardamalia and Bereiter, 2006)

Core design principles of OCL

Harasim emphasises the importance of three key phases of knowledge construction through discourse:

  • idea generating: this is literally brainstorming, to collect the divergent thinking within a group
  • idea organising: this is where learners compare, analyse and categorise the different ideas previously generated, again through discussion and argument
  • intellectual convergence: the aim here is to reach a level of intellectual synthesis, understanding and consensus (including agreeing to disagree), usually through the joint construction of some artefact or piece of work, such as an essay or assignment.

This results in what Harasim calls a Final Position, although in reality the position is never final because for a learner, once started, the process of generating, organising and converging on ideas continues at an ever deeper or more advanced level. The role of the teacher or instructor in this process is seen as critical, not only in facilitating the process and providing appropriate resources and learner activities that encourage this kind of learning, but also, as a representative of a knowledge community or subject domain, in ensuring that the core concepts, practices, standards and principles of the subject domain are fully integrated into the learning cycle. Harasim provides the following diagram to capture this process:

From Harasim (2012), p. 95

From Harasim (2012), Figure 6.3, p. 95

Figure 6. 1: Harasim’s pedagogy of group discussion

Another important factor is that in the OCL model, discussion forums are not an addition or supplement to core teaching materials, such as textbooks, recorded lectures, or text in an LMS, but are the core component of the teaching. Textbooks, readings and other resources are chosen to support the discussion, not the other way round. This is a key design principle, and explains why often instructors or tutors complain, in more ‘traditional’ online courses, that students don’t participate in discussions. Often this is because where online discussions are secondary to more didactic teaching, or are not deliberately designed and managed to lead to knowledge construction, students see the discussions as optional or extra work, because they have no direct impact on grades or assessment. (It is also a reason why awarding grades for participation in discussion forums misses the point. It is not the extrinsic activity that counts, but the intrinsic value of the discussion, that matters – see Brindley, Walti and Blashke, 2009). Thus although instructors using an OCL approach may use learning management systems for convenience, they are used differently from courses where traditional didactic teaching is moved online.

Community of Inquiry

The Community of Inquiry Model (CoI) is somewhat similar to the OCW model. As defined by Garrison, Anderson and Archer (2000)

An educational community of inquiry is a group of individuals who collaboratively engage in purposeful critical discourse and reflection to construct personal meaning and confirm mutual understanding.

Garrison, Anderson and Archer argue that there are three essential elements of a community of inquiry:

  • social presence ” is the ability of participants to identify with the community (e.g., course of study), communicate purposefully in a trusting environment, and develop inter-personal relationships by way of projecting their individual personalities.”
  • teaching presence  is “the design, facilitation, and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes
  • cognitive presence “is the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse“.
Image: © Marguerite Koole, 2013

Image: © Marguerite Koole, 2013

Other design principles

However, I consider CoI more of a theory than a model, since it does not indicate what activities or conditions are needed to create these three ‘presences’. I also see the two models as complementary rather than competing. Since the publication of the original CoI paper in 2000, there have been a number of studies that have identified the importance of these ‘presences’ within especially online learning (click here for a wide selection). Partly as a result of this research, and partly as the result of experienced online instructors who have not necessarily been influenced by either the OCL or the Community of Inquiry literature, several other design principles have been associated with successful (online) discussion, such as:

  • choice of appropriate technology (e.g. software that allows for threaded discussions)
  • clear guidelines on student online behaviour
  • student orientation and preparation, including technology orientation and explaining the purpose of discussion
  • choice of appropriate topics
  • setting an appropriate ‘tone’ or requirements for discussion (e.g. respectful disagreement, evidence-based arguments)
  • defining clearly learner roles and expectations
  • monitoring the participation of individual learners, and responding accordingly
  • regular, ongoing instructor ‘presence’
  • ensuring strong articulation between discussion topics and assessment.

These issues are discussed in more depth by Salmon (2000); Paloff and Pratt (2005; 2007); and Bates and Poole (2003).

Therefore, although there has been a wide range of researchers and educators engaged in the area of online collaborative learning and communities of inquiry, there is a high degree of convergence and agreement about successful strategies and design principles.

Strengths and weaknesses of online collaborative learning

This approach to the use of technology for teaching is very different from the more objectivist approaches found in computer-assisted learning, teaching machines, and artificial intelligence applications to education, which primarily aim to use computing to replace at least some of the activities traditionally done by human teachers. With online collaborative learning, the aim is not to replace the teacher, but to use the technology primarily to increase and improve communication between teacher and learners, with a particular approach to the development of learning based on knowledge construction assisted and developed through social discourse. This social discourse furthermore is not random, but managed in such a way as to ‘scaffold’ learning:

  • by assisting with the construction of knowledge in ways that are guided by the instructor,
  • that reflect the norms or values of the discipline, and
  • that also respect or take into consideration the prior knowledge within the discipline.

Thus there are two main strengths of this model:

  • when applied appropriately, online collaborative learning can lead to deep, academic learning, or transformative learning, as well as, if not better than, discussion in campus-based classrooms. The asynchronous and recorded ‘affordances’ of online learning more than compensate for the lack of physical cues and other aspects of face-to-face discussion.
  • online collaborative learning as a result can also directly support the development of a range of high level intellectual skills, such as critical thinking, analytical thinking, synthesis, and evaluation, which are key requirements for learners in a digital age.

There are though several limitations:

  • it does not scale easily, requiring highly knowledgeable and skilled instructors, and a limited number of learners
  • it is more likely to accommodate to the epistemological positions of faculty and instructors in humanities, social sciences, education and some areas of business studies and health and conversely it is likely to be less accommodating to the epistemological positions of faculty in science and engineering. However, if combined with a problem-based or inquiry-based approach, it might have acceptance even in some of these subject domains.

It could also be argued that there is no or little difference between online collaborative learning/Communities of Inquiry than well-conducted traditional classroom, discussion-based teaching. Once again, we see that the mode of delivery is less important than the design model, which can work well in both contexts. Indeed, it is possible to conduct either models synchronously or asynchronously, at a distance or face-to-face.

However, there is certainly enough evidence that collaborative learning can be done just as well online, which is important, given the need for more flexible models of delivery to meet the needs of a more diverse student body in a digital age. Also, the necessary conditions for success in teaching this way are now well known, even though they are not universally applied.

Over to you

I am of course inviting Linda, and the Albertans, to respond to this (well, it is Grey Cup tomorrow: Calgary, Alberta vs Hamilton, Ontario, in a curious game called Canadian Football, which is almost as good a dust-up). However, I’d also like comments from some less committed educators with regards to the following:

1. Can you see the differences between ‘Open Collaborative Learning’ (OCL) and ‘Communities of Inquiry’? Or are they really the same model with different names?

2. Do you agree that either of these models can be applied just as successfully online or face-to-face?

3. Do you see other strengths or weaknesses with these models?

4. Is this common sense dressed up as theory?

5. Has anyone had any experience of using either of these models in the quantitative sciences such as physics or engineering? If so, do you still have a job?

References

Bates, A. and Poole, G. (2003) Effective Teaching with Technology in Higher Education: Foundations for Success San Francisco: Jossey-Bass

Brindley, J., Walti, C. and Blashke, L. (2009) Creating Effective Collaborative Learning Groups in an Online Environment International Review of Research in Open and Distance Learning, Vol. 10, No. 3

Entwistle, N. (2000) Promoting deep learning through teaching and assessment: conceptual frameworks and educational contexts Leicester UK: TLRP Conference

Garrison, R., Anderson, A. and Archer, W. (2000) Critical Inquiry in a Text-based Environment: Computer Conferencing in Higher Education The Internet and Higher Education, Vol. 2, No. 3

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Laurillard, D. (2001) Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies New York/London: Routledge

Marton, F. and Saljö, R. (1997) Approaches to learning, in Marton, F., Hounsell, D. and Entwistle, N. (eds.) The experience of learning: Edinburgh: Scottish Academic Press (out of press, but available online)

Paloff, R. and Pratt, K. (2005) Collaborating Online: Learning Together in Community San Francisco: Jossey-Bass

Paloff, R. and Pratt, K. (2007) Building Online Learning Communities: Effective Strategies for the Virtual Classroom San Francisco: Jossey-Bass

Pask, G. (1975) Conversation, Cognition and Learning Amsterdam/London: Elsevier (out of press, but available online)

Salmon, G. (2000) e-Moderating: The Key to Teaching and Learning Online London: Taylor and Francis

Scardamalia, M. and Bereiter, C. (2006) Knowledge Building:  Theory, pedagogy and technology in Sawyer, K. (ed.) Cambridge Handbook of the Learning Sciences New York: cambridge University Press

 

The strengths and weaknesses of MOOCs: Part 2: learning and assessment

Remote exam proctoring

Remote exam proctoring

The writing of my book, Teaching in a Digital Age, has been interrupted for nearly two weeks by my trip to England for the EDEN Research Workshop. As part of the first draft of the book, I have already published three posts on MOOCs:

In this post, I ask (and try to answer) what do participants learn from MOOCs, and I also evaluate their assessment methods.

What do students learn in MOOCs?

This is a much more difficult question to answer, because so little of the research to date (2014) has tried to answer this question. (One reason, as we shall see, is that assessment of learning in MOOCs remains a major challenge). There are at least two kinds of study: quantitative studies that seek to quantify learning gains; and qualitative studies that describe the experience of learners within MOOCs, which indirectly provide some insight into what they have learned.

At the time of writing, the most well conducted study of learning in MOOCs has been by Colvin et al. (2014), who investigated ‘conceptual learning’ in an MIT Introductory Physics MOOC. They compared learner performance not only between different sub-categories of learners within the MOOC, such as those with no physics or math background with those such as physic teachers who had considerable prior knowledge, but also with on-campus students taking the same curriculum in a traditional campus teaching format. In essence, the study found no significant differences in learning gains between or within the two types of teaching, but it should be noted that the on-campus students were students who had failed an earlier version of the course and were retaking it.

This research is a classic example of the no significant difference in comparative studies in educational technology; other variables, such as differences in the types of students, were as important as the mode of delivery. Also, this MOOC design represents a behaviourist-cognitivist approach to learning that places heavy emphasis on correct answers to conceptual questions. It doesn’t attempt to develop the skills needed in a digital age as identified in Chapter 1.

There have been far more studies of the experience of learners within MOOCs, particularly focusing on the discussions within MOOCs (see for instance, Kop, 2011). In general (although there are exceptions), discussions are unmonitored, and it is left to participants to make connections and respond to other students comments.

However, there are some strong criticisms of the effectiveness of the discussion element of MOOCs for developing the high-level conceptual analysis required for academic learning. To develop deep, conceptual learning, there is a need in most cases for intervention by a subject expert, to clarify misunderstandings or misconceptions, to provide accurate feedback,  to ensure that the criteria for academic learning, such as use of evidence, clarity of argument, etc., are being met, and to ensure the necessary input and guidance to seek deeper understanding (see Harasim, 2013).

Furthermore, the more massive the course, the more likely participants are to feel ‘overload, anxiety and a sense of loss’, if there is not some instructor intervention or structure imposed (Knox, 2014). Firmin et al. (2014) have shown that when there is some form of instructor ‘encouragement and support of student effort and engagement’, results improve for all participants in MOOCs. Without a structured role for subject experts, participants are faced with a wide variety of quality in terms of comments and feedback from other participants. There is again a great deal of research on the conditions necessary for the successful conduct of collaborative and co-operative group learning (see for instance, Dillenbourg, 1999, Lave and Wenger, 1991), and these findings certainly have not been generally applied to the management of MOOC discussions to date.

One counter argument is that at least cMOOCs develop a new form of learning based on networking and collaboration that is essentially different from academic learning, and MOOCs are thus more appropriate to the needs of learners in a digital age. Adult participants in particular, it is claimed by Downes and Siemens, have the ability to self-manage the development of high level conceptual learning.  MOOCs are ‘demand’ driven, meeting the interests of individual students who seek out others with similar interests and the necessary expertise to support them in their learning, and for many this interest may well not include the need for deep, conceptual learning but more likely the appropriate applications of prior knowledge in new or specific contexts. MOOCs do appear to work best for those who already have a high level of education and therefore bring many of the conceptual skills developed in formal education with them when they join a MOOC, and therefore contribute to helping those who come without such skills.

Over time, as more experience is gained, MOOCs are likely to incorporate and adapt some of the findings from research on smaller group work for much larger numbers. For instance, some MOOCs are using ‘volunteer’ or community tutors (Dillenbourg, 2014).The US State Department has organized MOOC camps through US missions and consulates abroad to mentor MOOC participants. The camps include Fulbright scholars and embassy staff who lead discussions on content and topics for MOOC participants in countries abroad (Haynie, 2014). Some MOOC providers, such as the University of British Columbia, pay a small cohort of academic assistants to monitor and contribute to the MOOC discussion forums (Engle, 2014). Engle reported that the use of academic assistants, as well as limited but effective interventions from the instructors themselves, made the UBC MOOCs more interactive and engaging. However, paying for people to monitor and support MOOCs will of course increase the cost to providers. Consequently, MOOCs are likely to develop new automated ways to manage discussion effectively in very large groups. The University of Edinburgh is experimenting with automated ‘teacherbots’ that crawl through online discussion forums and direct predetermined comments to students identified as needing help or encouragement (Bayne, 2014).

These results and approaches are consistent with prior research on the importance of instructor presence for successful for-credit online learning. In the meantime, though, there is much work still to be done if MOOCs are to provide the support and structure needed to ensure deep, conceptual learning where this does not already exist in students. The development of the skills needed in a digital age is likely to be an even greater challenge when dealing with massive numbers. However, we need much more research into what participants actually learn in MOOCs and under what conditions before any firm conclusions can be drawn.

Assessment

Assessment of the massive numbers of participants in MOOCs has proved to be a major challenge. It is a complex topic that can be dealt with only briefly here. However, Chapter 5.8 provides a general analysis of different types of assessment, and Suen (2014) provides a comprehensive and balanced overview of the way assessment has been used in MOOCs to date. This section draws heavily on Suen’s paper.

Computer marked assignments

Assessment to date in MOOCs has been primarily of two kinds. The first is based on quantitative multiple-choice tests, or response boxes where formulae or ‘correct code’ can be entered and automatically checked. Usually participants are given immediate automated feedback on their answers, ranging from simple right or wrong answers to more complex responses depending on the type of response they have checked, but in all cases, the process is usually fully automated.

For straight testing of facts, principles, formulae, equations and other forms of conceptual learning where there are clear, correct answers, this works well. In fact, multiple choice computer marked assignments were used by the UK Open University as long ago as the 1970s, although the means to give immediate online feedback were not available then. However, this method of assessment is limited for testing deep or ‘transformative’ learning, and particularly weak for assessing the intellectual skills needed in a digital age, such as creative or original thinking.

Peer review

The second type of assessment that has been tried in MOOCs has been peer assessment, where participants assess each other’s work. Peer assessment is not new. It has been successfully used for formative assessment in traditional classrooms and in some online teaching for credit (Falchikov and Goldfinch, 2000; van Zundert et al., 2010). More importantly, peer assessment is seen as a powerful way to improve deep understanding and knowledge through the rating process, and at the same time, it can be useful for developing some of the skills needed in a digital age, such as critical thinking, for those participants assessing the work of others.

However, a key feature of the successful use of peer assessment has been the close involvement of an instructor or teacher, in providing benchmarks, rubrics or criteria  for assessment, and for monitoring and adjusting peer assessments to ensure consistency and a match with the benchmarks set by the instructor. Although an instructor can provide the benchmarks and rubrics in MOOCs, close monitoring of the multiple peer assessments is difficult if not impossible with the very large numbers of participants in MOOCs. As a result, MOOC participants often become incensed at being randomly assessed by other participants who may not and often do not have the knowledge or ability to give a ‘fair’ or accurate assessment of a participant’s work.

Various attempts to get round the limitations of peer assessment in MOOCs have been tried such as calibrated peer reviews, based on averaging all the peer ratings, and Bayesian post hoc stabilization, but although these statistical techniques reduce the error (or spread) of peer review somewhat they still do not remove the problems of systematic errors of judgement in raters due to misconceptions. This is particularly a problem where a majority of participants fail to understand key concepts in a MOOC, in which case peer assessment becomes the blind leading the blind.

Automated essay scoring

This is another area where there have been attempts to automate scoring. Although such methods are increasingly sophisticated they are currently limited in terms of accurate assessment to measuring primarily technical writing skills, such as grammar, spelling and sentence construction. Once again they do not measure accurately essays where higher level intellectual skills are demonstrated.

Badges and certificates

Particularly in xMOOCs, participants may be awarded a certificate or a ‘badge’ for successful completion of the MOOC, based on a final test (usually computer-marked) which measures the level of learning in a course. The American Council on Education (ACE), which represents the presidents of U.S. accredited, degree-granting institutions, recommended offering credit for five courses on the Coursera MOOC platform. However, according to the person responsible for the review process:

what the ACE accreditation does is merely accredit courses from institutions that are already accredited. The review process doesn’t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.’ (Book, 2013)

Indeed, most of the institutions offering MOOCs will not accept their own certificates for admission or credit within their own, campus-based programs. Probably nothing says more about the confidence in the quality of the assessment than this failure of MOOC providers to recognize their own teaching.

The intent behind assessment

To evaluate assessment in MOOCs requires an examination of the intent behind assessment. As identified earlier in another chapter of my book, there are many different purposes behind assessment. Peer assessment and immediate feedback on computer-marked tests can be extremely valuable for formative assessment, enabling participants to see what they have understood and to help develop further their understanding of key concepts. In cMOOCs, as Suen points out, learning is measured as the communication that takes place between MOOC participants, resulting in crowdsourced validation of knowledge – it’s what the sum of all the participants come to believe to be true as a result of participating in the MOOC, so formal assessment is unnecessary. However, what is learned in this way is not necessarily academically validated knowledge, which to be fair, is not the concern of cMOOC proponents such as Stephen Downes.

Academic assessment is a form of currency, related not only to measuring student achievement but also affecting student mobility (e.g. entrance to grad school) and perhaps more importantly employment opportunities and promotion. From a learner’s perspective, the validity of the currency – the recognition and transferability of the qualification – is essential. To date, MOOCs have been unable to demonstrate that they are able to assess accurately the learning achievements of participants beyond comprehension and knowledge of ideas, principles and processes (recognizing that there is some value in this alone). What MOOCs have not been able to demonstrate is that they can either develop or assess deep understanding or the intellectual skills required in a digital age. Indeed, this may not be possible within the constraints of massiveness, which is their major distinguishing feature from other forms of online learning, although the lack of valid methods of assessment will not stop computer scientists from trying to find ways to analyze participant online behaviour to show that such learning is taking place.

Up next

I hope the next post will be my last on this chapter on MOOCs. It will cover the following topics:

  • the cost of MOOCs and economies of scale
  • branding
  • the political, economic and social factors that explain the rise of MOOCs.

Over to you

As regular readers know, this is my way of obtaining peer review for my open textbook (so clearly I am not against peer review in principle!). So if I have missed anything important on this topic, or have misrepresented people’s views, or you just plain disagree with what I’ve written, please let me know. In particular, I am hoping for comments on:

  • comprehensiveness of the sources used that address learning and assessment methods in MOOCs
  • arguments that should have been included, either as a strength or a weakness
  • errors of fact

Yes, I’m a glutton for punishment, but you need to be a masochist to publish openly on this topic.

References

Bayne, S. (2014) Teaching, Research and the More-than-Human in Digital Education Oxford UK: EDEN Research Workshop (url to come)

Book, P. (2103) ACE as Academic Credit Reviewer–Adjustment, Accommodation, and Acceptance WCET Learn, July 25

Colvin, K. et al. (2014) Learning an Introductory Physics MOOC: All Cohorts Learn Equally, Including On-Campus Class, IRRODL, Vol. 15, No. 4

Dillenbourg, P. (ed.) (1999) Collaborative-learning: Cognitive and Computational Approaches. Oxford: Elsevier

Dillenbourg, P. (2014) MOOCs: Two Years Later, Oxford UK: EDEN Research Workshop (url to come)

Engle, W. (2104) UBC MOOC Pilot: Design and Delivery Vancouver BC: University of British Columbia

Falchikov, N. and Goldfinch, J. (2000) Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks Review of Educational Research, Vol. 70, No. 3

Firmin, R. et al. (2014) Case study: using MOOCs for conventional college coursework Distance Education, Vol. 35, No. 2

Haynie, D. (2014). State Department hosts ‘MOOC Camp’ for online learners. US News,January 20

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Ho, A. et al. (2014) HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013 (HarvardX and MITx Working Paper No. 1), January 21

Knox, J. (2014) Digital culture clash: ‘massive’ education in the e-Learning and Digital Cultures Distance Education, Vol. 35, No. 2

Kop, R. (2011) The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course International Review of Research into Open and Distance Learning, Vol. 12, No. 3

Lave, J. and Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press

Milligan, C., Littlejohn, A. and Margaryan, A. (2013) Patterns of engagement in connectivist MOOCs, Merlot Journal of Online Learning and Teaching, Vol. 9, No. 2

Suen, H. (2104) Peer assessment for massive open online courses (MOOCs) International Review of Research into Open and Distance Learning, Vol. 15, No. 3

van Zundert, M., Sluijsmans, D., van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20, 270-279