July 24, 2016

Choosing a ‘good’ post-secondary online learning program

Listen with webReader

woman at computer 2

I am constantly asked to recommend ‘good’ online learning programs. This is a difficult question to answer, as it’s impossible for any single person to know all the good online programs, and in any case, any selection is going to be highly subjective. However, it is possible to suggest a set of criteria or questions to help you in your decision, if you are thinking of taking an online program.

Criteria for selection

1. What prior qualifications do I need in order to take this program? Although courses may be online, they are not necessarily open. Generally high school completion is the minimum level of qualification for admission for a college or university online program, or a bachelor’s degree for graduate programs, except for open universities (see below).

2. What is the general status of the offering institution, for example, is it an accredited school with a generally good reputation?

3. Experience in online learning: how long has it been offering online programs? Institutions, like people, get better with practice.

4. The size of the operation: does it have many online courses, whole online programs, and lots of online students? If it does it is likely to have good systems in place to support online learning. On the other hand, if it’s grown very rapidly, it may have cut corners on quality.

5. Does any tenured or full-time instructor have overall responsibility for the course or program? If not, it is likely to be primarily a money-making operation for the institution.

6. What is the instructor:student ratio? Will students be taught by full-time faculty, adjunct faculty or teaching assistants? (I have no problem with adjunct faculty, so long as they are well qualified academically and responsible to a full-time faculty member, but beware of courses taught online by teaching assistants or unqualified ‘tutors’.)  Do instructors or faculty receive any training in teaching online? (This is especially important for contract or adjunct instructors.)

7. Do you personally know anything about the program? Have you studied any of their online courses – or on-campus courses for that matter? Do you know the people managing the online program?

8. What do students think about the program? What’s the completion rate? (Well designed online courses should have a successful completion rate of above 80%, and for a whole program, more than half of those who start the program should complete it.)

9. How expensive is it? Can I afford it? Will I be eligible for grants or student loans if I take this program?

10. What can I do with the qualification? Can I transfer the credits from an online course into an on-campus program? Will the qualification be recognised by any appropriate professional or accrediting agency?

My recommendation to anyone considering an online course or program is to apply the above criteria to any course or program you are interested in. Any self-respecting institution should be able to answer these questions, through the program web site, or by your calling the office that supports the program. Getting answers to these ten questions is more likely to enable you to find the right course than any recommendation I may make about individual courses or programs.

Personal recommendations

Nevertheless, I’m willing to stick my neck out, based mainly on my personal knowledge, but caveat emptor: you are responsible for your own decisions, so don’t blame me if one of these recommendations doesn’t work out for you. And just because I haven’t mentioned a particular institution doesn’t mean that it has poor quality programs – there are too many good programs to list them all.

1. Publicly funded dual mode institutions

These are publicly funded and accredited institutions that offer both on-campus and distance teaching, and usually have done so for many years. Typical examples are the land grant universities in the USA, which were established originally with state-wide responsibilities, e.g. Penn State’s World Campus, the University of Wisconsin System eCampus, and the University of New Mexico Online. These online courses and programs carry exactly the same weight as their on-campus courses. In Canada, similar institutions would be Memorial University, Laval University (in French), the University of Saskatchewan, the University of Manitoba, and the University of British Columbia.

These universities usually use their own ‘on-campus’ faculty combined with professional support staff such as instructional designers experienced in online and distance learning, although they usually also hire contract or adjunct instructors to support the main faculty instructor.

In Mexico, the Universidad de Guadalajara has an excellent online program in Spanish through its Virtual Campus.

There are many others, too numerous to list all of them, so check your local state or province’s public universities and then ask the ten questions above, as not all ‘dual-mode’ institutions have moved quickly enough away from print-based to fully online, so check that the course or program you are interested in is available online.

However, in general, choosing an online program from any university in this category is probably your safest bet in terms of quality and value.

2. Publicly funded universities that have specialised in online or blended learning.

These are universities or colleges that did not have much prior history of distance education but have moved extensively into online learning, both in blended and fully online formats. The best example in the USA is the University of Central Florida, Another with a strong online program is Empire State College in the State University of New York system. In Canada, the University of Guelph, the University of Ottawa (English and French), Laurentian University (English and French) and Queen’s University in Ontario, and Simon Fraser University in British Columbia, all have extensive online programs.

One of the largest online programs in the USA is the University of Maryland University College, which has many US servicemen as students. However, having been a guest tutor on some of their courses, I’m not particularly impressed with the overall quality, although some individual courses/programs are good.

In Canada, Royal Roads University in British Columbia offers a unique blended learning model consisting of one semester on campus and two or more semesters online, as well as some fully online programs, mainly at graduate level.

3. Publicly funded two-year community colleges

Many publicly funded community colleges in both the USA and Canada have excellent online programs. George Brown College and Algonquin College in Ontario have extensive online programs.

The Colorado Community College System offers a wide range of courses online, including science courses using home kits and remote labs. There are several other regional or state-wide online consortia in the USA, such as the Southern Regional Education Board’s Electronic Campus, covering 16 states.

Rio Salado Community College in Arizona has one of the largest online programs in the USA. However, there have been criticisms about the quality, because of its heavy reliance on short-term contract instructors who often lack training in online teaching. This is something to watch for in any institution that has suddenly and rapidly expanded its online programs.

3. Private, non-profit universities

The Western Governors University specializes in competency-based learning, which allows you to study at your own pace and take into account any prior learning or competencies, which means you may be able to complete a program more quickly. Usually programs are designed in consultation with major employers, ensuring acceptance of the degree. For more on the Western Governors model, click here.

The University of Southern New Hampshire has one of the largest online programs in the USA, and is also moving heavily into competency based learning.

Tec de Monterrey is a dual-mode private, non-profit university in Mexico that offers high quality online programs in Spanish.

4. Publicly funded open universities.

The best online is the Open University of Catalonia, in Spain, which was founded in 1996 as a purely online university. It offers several graduate programs in English, and many undergraduate as well as graduate programs in Spanish and Catalan. It accepts many international enrolments, particularly from Latin America.

Tèluq in Quebec offers high quality online programs in French.

The U.K. Open University and the Open University of the Netherlands are also leaders in online learning. The U.K. Open University consistently ranks in the top ten universities in the U.K. (out of over 180) in terms of teaching.

There are many other open universities around the world, some with hundreds of thousands of students (e.g. the Open University of China, Anadolu Open University in Turkey, the Open University in Indonesia, Indira Gandhi Open University in India, and the University of South Africa). However, because many students in these countries lack easy access to the Internet, most of these universities are still primarily print-based.

Indeed, most open universities have a heavy legacy of print-based education which has limited their ability to move completely online. I’m not currently recommending Athabasca University in Canada, because it’s future is uncertain, and many of its undergraduate programs are still print-based, although it has some excellent online graduate programs. The same applies to Thompson Rivers University’s Open Learning program in British Columbia.

Nevertheless, open universities offer many people their only chance of a higher education, especially in developing countries.

5. For-profit private universities

I hesitate to recommend for-profit institutions such as the University of Phoenix, or Kaplan, not because they don’t offer high quality online programs – they do – but because of ongoing problems with federal student aid and recognition of their degrees. For further discussion especially of the University of Phoenix, see ‘How does the University of Phoenix measure up?

6. MOOCs

If you are not looking for a qualification, but are just interested in a particular topic, there are many free, massive open online courses (MOOCs) available to you from many of the world’s leading universities. The following are the main platforms/sites where you can find such courses:

Open Education Europa provides a comprehensive list of MOOCs being offered by European institutions.

However, be warned: while certificates may be offered for successful course completion, these are not usually accepted by even the offering institutions towards a formal degree. However the OERu is building a free degree program on open educational resources.

7. General advice

It makes sense to take an online program from any local institution that you know well and trust. There are risks in taking online programs from out of state or out of country, unless the institution has an international reputation. There may also be language or cultural issues. It is particularly important that you get frequent and good quality interaction with an instructor on the program. Nevertheless the choice has never been so rich.

Over to you

There are many other universities and colleges offering excellent online programs, too many to number, but if you have experience of taking online programs and would like to make recommendations (positive or negative), please use the comment box below.

However, I will not publish any comments that are not offered in a thoughtful and constructive manner, and especially if they are being used solely to market (or trash) a particular program.

And any requests to list or mention the many commercial providers of online programs will be ignored.

 

Ensuring quality teaching in a digital age: key takeaways

Listen with webReader
Building the foundations of quality teaching and learning

Building the foundations of quality teaching and learning

I have now completed and published Chapter 11, ‘Ensuring quality teaching in a digital age‘, for my online open textbook, Teaching in a Digital Age.’

Unlike earlier chapters, I have not published this as a series of blog posts, as it is based on an earlier set of blog posts called: ‘Nine steps to quality online learning.’

However, there are some substantial changes. The focus here is as much on applying basic principles of course design to face-to-face and blended/hybrid learning as to fully online course design.

More importantly, this chapter attempts to pull together all the principles from all previous ten chapters into a set of practical steps towards the design of quality teaching in a digital age.

Purpose of the chapter

When you have read this chapter, and in conjunction with what has been learned in previous chapters, you should be able to:

  • define quality in terms of teaching in a digital age
  • determine what your preferred approaches are to teaching and learning
  • decide what mode of delivery is most appropriate for any course you are responsible for
  • understand why teamwork is essential for effective teaching in a digital age
  • make best use of existing resources for any course
  • choose and use the right technology and tools to support your learning
  • set appropriate learning goals for teaching in a digital age
  • design an appropriate course structure and set of learning activities
  • know when and how to communicate with learners
  • evaluate your teaching, make necessary improvements, and improve your teaching through further innovation.

What is covered in this chapter

Key takeaways

1. For the purposes of this book, quality is defined as: teaching methods that successfully help learners develop the knowledge and skills they will require in a digital age.

2. Formal national and institutional quality assurance processes do not guarantee quality teaching and learning. In particular, they focus on past ‘best’ practices, processes to be done before actual teaching, and often ignore the affective, emotional or personal aspects of learning. Nor do they focus particularly on the needs of learners in a digital age.

3. New technologies and the needs of learners in a digital age require a re-thinking of traditional campus-based teaching, especially where it is has been based mainly on the transmission of knowledge. This means re-assessing the way you teach and determining how you would really like to teach in a digital age. This requires imagination and vision rather than technical expertise.

4. It is important to determine the most appropriate mode of delivery, based on teaching philosophy, the needs of students, the demands of the discipline, and the resources available.

5. It is best to work in a team. Blended and especially fully online learning require a range of skills that most instructors are unlikely to have. Good course design not only enables students to learn better but also controls teacher and instructor workload. Courses look better with good graphic and web design and professional video production. Specialist technical help frees up teachers and instructors to concentrate on the knowledge and skills that students need to develop.

6. Full use should be made of existing resources, including institutionally-supported learning technologies, open educational resources, learning technology staff, and the experience of your colleagues.

7. The main technologies you will be using should be mastered, so you are professional and knowledgeable about their strengths and weaknesses for teaching.

8. Learning goals that are appropriate for learners in a digital age need to be clearly defined. The skills students need should be embedded within their subject domain, and these skills should be formally assessed.

9. A coherent and clearly communicable structure, and learning activities for a course, should be developed that are manageable in terms of workload for both students and instructor.

10. Regular and on-going instructor/teacher presence, especially when students are studying partly or wholly online, is essential for student success. This means effective communication between teacher/instructor and students. It is particularly important to encourage inter-student communication, either face-to-face or online.

11. The extent to which the new learning goals of re-designed courses aimed at developing the knowledge and skills needed in a digital age have been achieved should be carefully evaluated and ways in which the course could be improved should be identified.

Over to you

Although the previous blog posts on nine steps to quality online learning were well received (they have been used in some post-secondary education courses) feedback on this revised book version will be much appreciated.  I haven’t seen anything similar that tries to integrate basic principles across all three modes of delivery, so I am especially interested to see how these are perceived in terms of regular classroom and blended learning.

Up next

The final chapter, which will take a brief look at the institutional policies and strategies needed to support teachers and instructors wanting to teach well in a digital age. It will deal explicitly with what we should expect (and more importantly, not expect) of teachers and instructors, issues around faculty development and teacher training, working methods for teachers and instructors, and learning technology support.

I aim to finish this (and the whole book, at least in first draft form) by March 14. French and Spanish translations are already under way.

A future vision for OER and online learning

Listen with webReader

For each chapter of my online open textbook, Teaching in a Digital Age, I am developing imaginary but hopefully realistic scenarios. In this scenario, developed as a closing to my chapter on ‘Modes of Delivery and Open Education’, I look at how modularization could lead both a wider range of access to credit courses and more open use of learning materials.

Print

Figure 10.1 The Hart River, Yukon. Image: © www.protectpeel.ca, CC BY-NC

Figure 10.1.F The Hart River, Yukon.
Image: © www.protectpeel.ca, CC BY-NC

Print

Research faculty in the Faculties of Land Management and Forestry at the (mythical) University of Western Canada developed over a number of years a range of ‘learning artefacts’, digital graphics, computer models and simulations about watershed management, partly as a consequence of research conducted by faculty, and partly to generate support and funding for further research.

At a faculty meeting several years ago, after a somewhat heated discussion, faculty members voted to make these resources openly available for re-use for educational purposes under a Creative Commons license that requires attribution and prevents commercial use without specific written permission from the copyright holders, who in this case are the faculty responsible for developing the artefacts. What swayed the vote is that the majority of the faculty actively involved in the research wanted to make these resources more widely available. The agencies responsible for funding the work that lead to the development of the artefacts (mainly national research councils) welcomed the move to makes these artefacts more widely available as open educational resources.

Initially, the researchers just put the graphics and simulations up on the research group’s web site. It was left to individual faculty members to decide whether to use these resources in their teaching. Over time, faculty started to introduce these resources into a range of on-campus undergraduate and graduate courses.

After a while, though, word seemed to get out about these OER. The research faculty began to receive e-mails and phone calls from other researchers around the world. It became clear that there was a network or community of researchers in this field who were creating digital materials as a result of their research, and it made sense to share and re-use materials from other sites. This eventually led to an international web ‘portal’ of learning artefacts on watershed management.

The researchers also started to get calls from a range of different agencies, from government ministries or departments of environment, local environmental groups, First Nations/aboriginal bands, and, occasionally, major mining or resource extraction companies, leading to some major consultancy work for the faculty in the department. At the same time, the faculty were able to attract further research funding from non-governmental agencies such as the Nature Conservancy and some ecological groups, as well as from their traditional funding source, the national research councils, to develop more OER.

By this time, instructors had access to a fairly large amount of OER. There were already two fourth and fifth level fully online courses built around the OER that were being offered successfully to undergraduate and graduate students.

A proposal was therefore put forward to create initially a fully online post-graduate certificate program on watershed management, built around existing OER, in partnership with a university in the USA and another one in Sierra Leone. This certificate program was to be self-funding from tuition fees, with the tuition fees for the 25 Sierra Leone students to be initially covered by an international aid agency. The Dean, after a period of hard negotiation, persuaded the university administration that the tuition fees from the certificate program should go directly to the two Faculties whose staff were teaching the program.From these funds, the departments would hire additional tenured faculty to teach or backfill for the certificate, and the Faculties would pay 25 per cent of the tuition revenues to the university as overheads.

This decision was made somewhat easier by a fairly substantial grant from Foreign Affairs Canada to make the certificate program available in English and French to Canadian mining and resource extraction companies with contracts and partnerships in African countries.

Although the certificate program was very successful in attracting students from North America, Europe and New Zealand, it was not taken up very well in Africa beyond the partnership with the university in Sierra Leone, although there was a lot of interest in the OER and the issues raised in the certificate courses. After two years of running the certificate, then, the Faculties made two major decisions:

  • another three courses and a research project would be added to the certificate courses, and this would be offered as a fully cost recoverable online master in land and water systems. This would attract greater participation from managers and professionals in African countries in particular, and provide a recognised qualification that many of the certificate students were requesting
  • drawing on the large network of external experts now involved one way or another with the researchers, the university would offer a series of MOOCs on watershed management issues, with volunteer experts from outside the university being invited to participate and provide leadership in the MOOCs. The MOOCs would be able to draw on the existing OER.

Five years later, the following outcomes were recorded by the Dean of one of the faculties at an international conference on sustainability:

  • the online master’s program had doubled the total number of graduate students across the two faculties
  • the master’s program was fully cost-recoverable from tuition fees
  • there were 120 graduates a year from the master’s program
  • the degree completion rate was 64 per cent
  • six new tenured faculty has been hired, plus another six post-doctoral research faculty
  • several thousand students had registered and paid for at least one course in the certificate or master’s program, of which 45 per cent were from outside Canada
  • over 100,000 students had taken the MOOCs, almost half from developing countries
  • there were now over 1,000 hours of OER on watershed management available and downloaded many times across the world, attracting more students and revenue to the university
  • the university was now internationally recognised as a world leader in watershed management.

Although this scenario is purely a figment of my imagination, it is influenced by real and exciting work, much of which was developed as open access materials from the start, at the University of British Columbia:

Over to you

1. Does this strike you as a realistic scenario?

2. How useful are scenarios like this for thinking about the future? Could you use similar kinds of scenarios in your program planning or for faculty development, for instance?

3, If you have used scenarios for online learning in similar ways, would you be willing to share one?

4. Most of the elements of this scenario already exist at UBC. What I have done though is bring things together from different parts of the university into an integrated single scenario. What could be done within institutions to make this cross-disciplinary transfer of ideas and strategies easier to achieve? (It should be noted that UBC already has a Flexible Learning initiative, including a strategy team within the Provost’s office, which should help with this.)

Next

Just one more post to wrap up the chapter on Modes of Delivery and Open Education: the key takeaways from this chapter.

 

 

EDEN research papers: OERs (inc. MOOCs), quality/assessment, social media, analytics and research methods

Listen with webReader

EDEN RSW me 2

EDEN has now published a second report on my review of papers submitted to the EDEN research workshop in Oxford a couple of weeks ago. All the full papers for the workshop can be accessed here.

Main lessons (or unanswered questions) I took away:

OERs and MOOCs

  • what does awarding badges of certificates for MOOCs or other OER actually mean? For instance will institutions give course exemption or credits for the awards, or accept such awards for admission purposes? Or will the focus be on employer recognition? How will participants who are awarded badges know what their ‘currency’ is worth?
  • can MOOCs be designed to go beyond comprehension or networking to develop other critical 21st century skills such as critical thinking, analysis and evaluation? Can they lead to ‘transformational learning’ as identified by Kumar and Arnold (see Quality and Assessment below)
  • are there better design models for open courses than MOOCs as currently structured? If so what would they look like?
  • is there a future for learning object repositories when nearly all academic content becomes open and online?

Quality and assessment

  • research may inform but won’t resolve policy issues
  • quality is never ‘objective’ but is value-driven
  • the level of intervention must be long and significant enough to result in significant learning gains
  • there’s lots of research already that indicates the necessary conditions for successful use of online discussion forums but if these conditions are not present then learning will not take place
  • the OU’s traditional model of course design constrains the development of successful collaborative online learning.

Use of social media in open and distance learning

There were surprisingly few papers on this topic. My main takeaway:

  • the use of social media needs to be driven by sound pedagogical theory that takes into account the affordances of social media (as in Sorensen’s study described in an earlier post under course design)

Data analytics and student drop-out

  • institutions/registrars must pay attention to how student data is tagged/labeled for analytic purposes, so there is consistency in definitions, aggregation and interpretation;
  • when developing or applying an analytics software program, consideration needs to be given to the level of analysis and what potential users of the data are looking for; this means working with instructional designers, faculty and administrators from the beginning
  • analytics need to be integrated with action plans to identify and support early at risk students

Research methods

Next

If these bullets interest you at all, then I strongly recommend you go and read the original papers in full – click here. My summary is of necessity personal and abbreviated and the papers provide much greater richness of context.

 

 

The strengths and weaknesses of MOOCs: Part 2: learning and assessment

Listen with webReader
Remote exam proctoring

Remote exam proctoring

The writing of my book, Teaching in a Digital Age, has been interrupted for nearly two weeks by my trip to England for the EDEN Research Workshop. As part of the first draft of the book, I have already published three posts on MOOCs:

In this post, I ask (and try to answer) what do participants learn from MOOCs, and I also evaluate their assessment methods.

What do students learn in MOOCs?

This is a much more difficult question to answer, because so little of the research to date (2014) has tried to answer this question. (One reason, as we shall see, is that assessment of learning in MOOCs remains a major challenge). There are at least two kinds of study: quantitative studies that seek to quantify learning gains; and qualitative studies that describe the experience of learners within MOOCs, which indirectly provide some insight into what they have learned.

At the time of writing, the most well conducted study of learning in MOOCs has been by Colvin et al. (2014), who investigated ‘conceptual learning’ in an MIT Introductory Physics MOOC. They compared learner performance not only between different sub-categories of learners within the MOOC, such as those with no physics or math background with those such as physic teachers who had considerable prior knowledge, but also with on-campus students taking the same curriculum in a traditional campus teaching format. In essence, the study found no significant differences in learning gains between or within the two types of teaching, but it should be noted that the on-campus students were students who had failed an earlier version of the course and were retaking it.

This research is a classic example of the no significant difference in comparative studies in educational technology; other variables, such as differences in the types of students, were as important as the mode of delivery. Also, this MOOC design represents a behaviourist-cognitivist approach to learning that places heavy emphasis on correct answers to conceptual questions. It doesn’t attempt to develop the skills needed in a digital age as identified in Chapter 1.

There have been far more studies of the experience of learners within MOOCs, particularly focusing on the discussions within MOOCs (see for instance, Kop, 2011). In general (although there are exceptions), discussions are unmonitored, and it is left to participants to make connections and respond to other students comments.

However, there are some strong criticisms of the effectiveness of the discussion element of MOOCs for developing the high-level conceptual analysis required for academic learning. To develop deep, conceptual learning, there is a need in most cases for intervention by a subject expert, to clarify misunderstandings or misconceptions, to provide accurate feedback,  to ensure that the criteria for academic learning, such as use of evidence, clarity of argument, etc., are being met, and to ensure the necessary input and guidance to seek deeper understanding (see Harasim, 2013).

Furthermore, the more massive the course, the more likely participants are to feel ‘overload, anxiety and a sense of loss’, if there is not some instructor intervention or structure imposed (Knox, 2014). Firmin et al. (2014) have shown that when there is some form of instructor ‘encouragement and support of student effort and engagement’, results improve for all participants in MOOCs. Without a structured role for subject experts, participants are faced with a wide variety of quality in terms of comments and feedback from other participants. There is again a great deal of research on the conditions necessary for the successful conduct of collaborative and co-operative group learning (see for instance, Dillenbourg, 1999, Lave and Wenger, 1991), and these findings certainly have not been generally applied to the management of MOOC discussions to date.

One counter argument is that at least cMOOCs develop a new form of learning based on networking and collaboration that is essentially different from academic learning, and MOOCs are thus more appropriate to the needs of learners in a digital age. Adult participants in particular, it is claimed by Downes and Siemens, have the ability to self-manage the development of high level conceptual learning.  MOOCs are ‘demand’ driven, meeting the interests of individual students who seek out others with similar interests and the necessary expertise to support them in their learning, and for many this interest may well not include the need for deep, conceptual learning but more likely the appropriate applications of prior knowledge in new or specific contexts. MOOCs do appear to work best for those who already have a high level of education and therefore bring many of the conceptual skills developed in formal education with them when they join a MOOC, and therefore contribute to helping those who come without such skills.

Over time, as more experience is gained, MOOCs are likely to incorporate and adapt some of the findings from research on smaller group work for much larger numbers. For instance, some MOOCs are using ‘volunteer’ or community tutors (Dillenbourg, 2014).The US State Department has organized MOOC camps through US missions and consulates abroad to mentor MOOC participants. The camps include Fulbright scholars and embassy staff who lead discussions on content and topics for MOOC participants in countries abroad (Haynie, 2014). Some MOOC providers, such as the University of British Columbia, pay a small cohort of academic assistants to monitor and contribute to the MOOC discussion forums (Engle, 2014). Engle reported that the use of academic assistants, as well as limited but effective interventions from the instructors themselves, made the UBC MOOCs more interactive and engaging. However, paying for people to monitor and support MOOCs will of course increase the cost to providers. Consequently, MOOCs are likely to develop new automated ways to manage discussion effectively in very large groups. The University of Edinburgh is experimenting with automated ‘teacherbots’ that crawl through online discussion forums and direct predetermined comments to students identified as needing help or encouragement (Bayne, 2014).

These results and approaches are consistent with prior research on the importance of instructor presence for successful for-credit online learning. In the meantime, though, there is much work still to be done if MOOCs are to provide the support and structure needed to ensure deep, conceptual learning where this does not already exist in students. The development of the skills needed in a digital age is likely to be an even greater challenge when dealing with massive numbers. However, we need much more research into what participants actually learn in MOOCs and under what conditions before any firm conclusions can be drawn.

Assessment

Assessment of the massive numbers of participants in MOOCs has proved to be a major challenge. It is a complex topic that can be dealt with only briefly here. However, Chapter 5.8 provides a general analysis of different types of assessment, and Suen (2014) provides a comprehensive and balanced overview of the way assessment has been used in MOOCs to date. This section draws heavily on Suen’s paper.

Computer marked assignments

Assessment to date in MOOCs has been primarily of two kinds. The first is based on quantitative multiple-choice tests, or response boxes where formulae or ‘correct code’ can be entered and automatically checked. Usually participants are given immediate automated feedback on their answers, ranging from simple right or wrong answers to more complex responses depending on the type of response they have checked, but in all cases, the process is usually fully automated.

For straight testing of facts, principles, formulae, equations and other forms of conceptual learning where there are clear, correct answers, this works well. In fact, multiple choice computer marked assignments were used by the UK Open University as long ago as the 1970s, although the means to give immediate online feedback were not available then. However, this method of assessment is limited for testing deep or ‘transformative’ learning, and particularly weak for assessing the intellectual skills needed in a digital age, such as creative or original thinking.

Peer review

The second type of assessment that has been tried in MOOCs has been peer assessment, where participants assess each other’s work. Peer assessment is not new. It has been successfully used for formative assessment in traditional classrooms and in some online teaching for credit (Falchikov and Goldfinch, 2000; van Zundert et al., 2010). More importantly, peer assessment is seen as a powerful way to improve deep understanding and knowledge through the rating process, and at the same time, it can be useful for developing some of the skills needed in a digital age, such as critical thinking, for those participants assessing the work of others.

However, a key feature of the successful use of peer assessment has been the close involvement of an instructor or teacher, in providing benchmarks, rubrics or criteria  for assessment, and for monitoring and adjusting peer assessments to ensure consistency and a match with the benchmarks set by the instructor. Although an instructor can provide the benchmarks and rubrics in MOOCs, close monitoring of the multiple peer assessments is difficult if not impossible with the very large numbers of participants in MOOCs. As a result, MOOC participants often become incensed at being randomly assessed by other participants who may not and often do not have the knowledge or ability to give a ‘fair’ or accurate assessment of a participant’s work.

Various attempts to get round the limitations of peer assessment in MOOCs have been tried such as calibrated peer reviews, based on averaging all the peer ratings, and Bayesian post hoc stabilization, but although these statistical techniques reduce the error (or spread) of peer review somewhat they still do not remove the problems of systematic errors of judgement in raters due to misconceptions. This is particularly a problem where a majority of participants fail to understand key concepts in a MOOC, in which case peer assessment becomes the blind leading the blind.

Automated essay scoring

This is another area where there have been attempts to automate scoring. Although such methods are increasingly sophisticated they are currently limited in terms of accurate assessment to measuring primarily technical writing skills, such as grammar, spelling and sentence construction. Once again they do not measure accurately essays where higher level intellectual skills are demonstrated.

Badges and certificates

Particularly in xMOOCs, participants may be awarded a certificate or a ‘badge’ for successful completion of the MOOC, based on a final test (usually computer-marked) which measures the level of learning in a course. The American Council on Education (ACE), which represents the presidents of U.S. accredited, degree-granting institutions, recommended offering credit for five courses on the Coursera MOOC platform. However, according to the person responsible for the review process:

what the ACE accreditation does is merely accredit courses from institutions that are already accredited. The review process doesn’t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.’ (Book, 2013)

Indeed, most of the institutions offering MOOCs will not accept their own certificates for admission or credit within their own, campus-based programs. Probably nothing says more about the confidence in the quality of the assessment than this failure of MOOC providers to recognize their own teaching.

The intent behind assessment

To evaluate assessment in MOOCs requires an examination of the intent behind assessment. As identified earlier in another chapter of my book, there are many different purposes behind assessment. Peer assessment and immediate feedback on computer-marked tests can be extremely valuable for formative assessment, enabling participants to see what they have understood and to help develop further their understanding of key concepts. In cMOOCs, as Suen points out, learning is measured as the communication that takes place between MOOC participants, resulting in crowdsourced validation of knowledge – it’s what the sum of all the participants come to believe to be true as a result of participating in the MOOC, so formal assessment is unnecessary. However, what is learned in this way is not necessarily academically validated knowledge, which to be fair, is not the concern of cMOOC proponents such as Stephen Downes.

Academic assessment is a form of currency, related not only to measuring student achievement but also affecting student mobility (e.g. entrance to grad school) and perhaps more importantly employment opportunities and promotion. From a learner’s perspective, the validity of the currency – the recognition and transferability of the qualification – is essential. To date, MOOCs have been unable to demonstrate that they are able to assess accurately the learning achievements of participants beyond comprehension and knowledge of ideas, principles and processes (recognizing that there is some value in this alone). What MOOCs have not been able to demonstrate is that they can either develop or assess deep understanding or the intellectual skills required in a digital age. Indeed, this may not be possible within the constraints of massiveness, which is their major distinguishing feature from other forms of online learning, although the lack of valid methods of assessment will not stop computer scientists from trying to find ways to analyze participant online behaviour to show that such learning is taking place.

Up next

I hope the next post will be my last on this chapter on MOOCs. It will cover the following topics:

  • the cost of MOOCs and economies of scale
  • branding
  • the political, economic and social factors that explain the rise of MOOCs.

Over to you

As regular readers know, this is my way of obtaining peer review for my open textbook (so clearly I am not against peer review in principle!). So if I have missed anything important on this topic, or have misrepresented people’s views, or you just plain disagree with what I’ve written, please let me know. In particular, I am hoping for comments on:

  • comprehensiveness of the sources used that address learning and assessment methods in MOOCs
  • arguments that should have been included, either as a strength or a weakness
  • errors of fact

Yes, I’m a glutton for punishment, but you need to be a masochist to publish openly on this topic.

References

Bayne, S. (2014) Teaching, Research and the More-than-Human in Digital Education Oxford UK: EDEN Research Workshop (url to come)

Book, P. (2103) ACE as Academic Credit Reviewer–Adjustment, Accommodation, and Acceptance WCET Learn, July 25

Colvin, K. et al. (2014) Learning an Introductory Physics MOOC: All Cohorts Learn Equally, Including On-Campus Class, IRRODL, Vol. 15, No. 4

Dillenbourg, P. (ed.) (1999) Collaborative-learning: Cognitive and Computational Approaches. Oxford: Elsevier

Dillenbourg, P. (2014) MOOCs: Two Years Later, Oxford UK: EDEN Research Workshop (url to come)

Engle, W. (2104) UBC MOOC Pilot: Design and Delivery Vancouver BC: University of British Columbia

Falchikov, N. and Goldfinch, J. (2000) Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks Review of Educational Research, Vol. 70, No. 3

Firmin, R. et al. (2014) Case study: using MOOCs for conventional college coursework Distance Education, Vol. 35, No. 2

Haynie, D. (2014). State Department hosts ‘MOOC Camp’ for online learners. US News,January 20

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Ho, A. et al. (2014) HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013 (HarvardX and MITx Working Paper No. 1), January 21

Knox, J. (2014) Digital culture clash: ‘massive’ education in the e-Learning and Digital Cultures Distance Education, Vol. 35, No. 2

Kop, R. (2011) The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course International Review of Research into Open and Distance Learning, Vol. 12, No. 3

Lave, J. and Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press

Milligan, C., Littlejohn, A. and Margaryan, A. (2013) Patterns of engagement in connectivist MOOCs, Merlot Journal of Online Learning and Teaching, Vol. 9, No. 2

Suen, H. (2104) Peer assessment for massive open online courses (MOOCs) International Review of Research into Open and Distance Learning, Vol. 15, No. 3

van Zundert, M., Sluijsmans, D., van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20, 270-279