October 26, 2014

Special edition on research on MOOCs in the journal ‘Distance Education’

Listen with webReader
The University of Toronto is one of a number of institutions conducting research on MOOCs

The University of Toronto is one of a number of institutions conducting research on MOOCs; their results are still to come

The August 2014 edition of the Australian-based journal, Distance Education (Vol.35, No. 2.), is devoted to new research on MOOCs. There is a guest editor, Kemi Jona, from Northwestern University, Illinois, as well as the regular editor, Som Naidu.

The six articles in this edition are fascinating, both in terms of their content, but even more so in their diversity. There are also three commentaries, by Jon Baggaley, Gerhard Fischer and myself.

My commentary provides my personal analysis of the six articles.

MOOCs are a changing concept

In most of the literature and discussion about MOOCs, there is a tendency to talk about ‘instructionist’ MOOCs (i.e. Coursera, edX, Udacity, xMOOCs) or ‘connectivist’ MOOCs (i.e. Downes, Siemens, Cormier, cMOOCs). Although this is still a useful distinction, representing very different pedagogies and approaches, the articles in this edition show that MOOCs come in all sizes and varieties.

Indeed, it is clear that the design of MOOCs is undergoing rapid development, partly as a result of more players coming in to the market, partly because of the kinds of research now being conducted on MOOCs themselves, and, sadly much more slowly, a recognition by some of the newer players that much is already known about open and online education that needs to be applied to the design of MOOCs, while accepting that there are certain aspects, in particular the scale, that make MOOCs unique.

The diversity of MOOC designs

These articles illustrate clearly such developments. The MOOCs covered by the articles range from

  • MOOC video recorded lectures watched in isolation by learners (Adams et al.)
  • MOOC video lectures watched in co-located groups in a flipped classroom mode without instructor or tutorial support (Nan Li et al.)
  • MOOCs integrated into regular campus-based programs with some learner support (Firmin et al.)
  • MOOCs using participatory and/or connectivist pedagogy (Anderson, Knox)

Also the size of the different MOOC populations studied here differed enormously, from 54 students per course to 42,000.

It is also clear that MOOC material is being increasingly extracted from the ‘massive’, open context and used in very specific ‘closed’ contexts, such as flipped classrooms, at which point one questions the difference between such use of MOOCs and regular for-credit online programming, which in many cases also use recorded video lectures or online discussion and increasingly other sources of open educational materials. I would expect in such campus-based contexts the same quality standards to apply to the MOOC+ course designs as are already applied to credit-based online learning. Some of the research findings in these articles indirectly support the need for this.

The diversity of research questions on MOOCs

Almost as interesting is the range of questions covered by these articles, which include:

  • capturing the lived experience of being in a MOOC (Adams et al.; Knox)
  • the extent to which learners can/should create their own content, and the challenges around that (Knox; Andersen)
  • how watching video lectures in a group affects learner satisfaction (Nan Li et al.)
  • what ‘massive’ means in terms of a unique pedagogy (Knox)
  • the ethical implications of MOOCs (Marshall)
  • reasons for academic success and failure in ‘flipped’ MOOCs (Firmin et al.; Knox)

What is clear from the articles is that MOOCs raise some fundamental questions about the nature of learning in digital environments. In particular, the question of the extent to which learners need guidance and support in MOOCs, and how this can best be provided, were common themes across several of the papers, with no definitive answers.

The diversity of methodology in MOOC research

Not surprisingly, given the range of research questions, there is also a very wide range of methodologies used in the articles in this edition, ranging from

  • phenomenology (Adams),
  • heuristics (Marshall)
  • virtual ethnography (Knox; Andersen)
  • quasi-experimental comparisons (Nan Li et al.)
  • data and learning analytics (Firmin et al.)

The massiveness of MOOCs, their accessibility, and the wide range of questions they raise make the topic a very fertile area for research, and this is likely to generate new methods of research and analysis in the educational field.

Lessons learned

Readers are likely to draw a variety of conclusions from these studies. Here are mine:

  • the social aspect of learning is extremely important, and MOOCs offer great potential for exploiting this kind of learning, but organizing and managing social learning on a massive scale, without losing the potential advantages of collaboration at scale, is a major challenge that still remains to be adequately addressed. The Knox article in particular describes in graphic detail the sense of being overwhelmed by information in open connectivist MOOCs. We still lack methods or designs that properly support participants in such environments. This is a critical area for further research and development.
  • a lecture on video is still a lecture, whether watched in isolation or in groups. The more we attempt to support this transmissive model through organized group work, ‘facilitators’, or ‘advisors’ the closer we move towards conventional (and traditional) education and the further away from the core concept of a MOOC.
  • MOOCs have a unique place in the educational ecology. MOOCs are primarily instruments for non-formal learning. Trying to adapt MOOCs to the campus not only undermines their primary purpose, but risks moving institutions in the wrong direction. We would be better re-designing our large lecture classes from scratch, using criteria, methods and standards appropriate to the goals of formal higher education. My view is that in the long run, we will learn more from MOOCs about handling social learning at scale than about transmitting information at scale. We already know about that. It’s called broadcasting.
  • lastly, there was surprisingly little in the articles about what actual learning took place. In some cases, it was a deliberate research strategy not to enquire into this, relying more on student or instructor feelings and perceptions. While other potential benefits, such as institutional branding, stimulating interest, providing a network of connections, and so on, are important, the question remains: what are participants actually learning from MOOCs, and does this justify the hype and investment (both institutionally and in participants’ time) that surrounds them?

Cultural and ethical issues

The Marshall paper provides an excellent overview of ethical issues, but there is almost no representation of perspectives on MOOCs from outside Western contexts. I would have liked to have seen more on cultural and ethical issues arising from the globalization of MOOCs, based on actual cases or examples. Given the global implications of MOOCs, other perspectives are needed. Perhaps this is a topic for another issue.

Happy reading

I am sure you will be as fascinated and stimulated by these articles as I am. I am also sure you will come away with different conclusions from mine. I am sure we will see a flood of other articles soon on this topic. Nevertheless, these articles are important in setting the research agenda, and should be essential reading for MOOC designers as well as future researchers on this topic.

How to get the articles

To obtain access to these articles, go to: http://www.tandfonline.com/toc/cdie20/current#.U-1WqrxdWh1

The success or otherwise of online students in the California Community College system

Listen with webReader

 Online offerings vary widely across subject

Johnson, H. and Mejia, M. (2014) Online learning and student outcomes in California’s community colleges San Francisco CA: Public Policy Institute of California, 20 pp

I’m not a great fan of studies into completion rates in online learning, because most studies fail to take into account a whole range of factors outside of the mode of delivery that influence student outcomes. However, this study is an exception. Conducted by researchers at the highly influential PPIC, it takes a very careful look at how well students across the whole California community college system (CCCS) do in online learning, and there are some very interesting findings that may not come as a surprise to experienced observers of online learning, but will certainly provide fodder for both supporters and skeptics of online learning.

Why the study is important

Several reasons:

  • California’s community colleges offer more online credit courses than any other public higher education institution in the country. By 2012, online course enrollment in the state’s community colleges totaled almost one million, representing about 11 percent of total enrollment
  • Over the past ten years, online course enrollment has increased by almost 850,000, while traditional course enrollment has declined by almost 285,000.
  • Community colleges are more likely than other institutions of higher education [in the USA] to serve nontraditional students. These students often have employment and family obligations and therefore may potentially benefit the most from online learning.
  • The state of California is investing $57 million over the next five to six years for online learning initiatives within the California Community College system
  • The California Community Colleges Chancellor’s Office (CCCCO) provided … access to unique longitudinal student- and course-level data from all of the state’s 112 community colleges

Main findings

  • Between 2008–09 and 2011–12, total credit enrollment at California’s community colleges declined by almost a million. The scarcity of traditional courses has been a factor in the huge increase in online enrollments. With the state cutting support to community colleges by more than $1.5 billion between 2007–08 and 2011–12, community colleges experienced an unprecedented falloff in enrollment 
  • online course success rates are between 11 and 14 percentage points lower than traditional course success rates.
  • in the long term, students who take online classes tend to be more successful than those who enroll only in traditional courses…students who take at least some online courses are more likely than those who take only traditional courses to earn an associate’s degree or to transfer to a four-year institution.
  • for students juggling school, family and work obligations, the ability to maintain a full-time load by mixing in one or two online courses per term may outweigh the lower chances of succeeding in each particular online course.
  • if a student’s choice is between taking an online course or waiting for the course to be offered in a classroom at a convenient time, taking the online course can help expedite completion or transfer
  • participation in online courses has increased for each of the state’s largest ethnic groups—and online enrollment rates for African American students, an underrepresented group in higher education in California, are particularly high. However, these rates are much lower among Latino students.

Main recommendations

  • move from ad hoc offerings to more strategic planning of online courses
  • improve the ability to transfer credits between community colleges and between colleges and the state’s universities
  • improve the design and provide more consistency in the quality of online courses between institutions
  • adopt a standardized learning management system across all colleges
  • collect systematic information on the cost of developing and maintaining online courses

My comments

This is another excellent and succinct research report on online learning, with a very strong methodology and important results, even if I am not at all surprised by the outcomes. I would expect online completion rates for individual courses to be lower than for traditional courses as students taking online courses often have a wider range of other commitments to manage than full-time, on campus students.

Similarly, I’m not surprised that online course success is lightly lower for community colleges than for universities (if we take both the figures from Ontario and my own experience as a DE director) and for certain ethnic groups who suffer from a range of socio-economic disadvantages. Online learning is more demanding and requires more experience in studying. Post-graduate students tend to do better at online learning than undergraduate students, and final year undergraduate students tend to do better than first year undergraduate students. Nevertheless, as the study clearly indicates, over the long term online learning provides not only increased access but also a greater chance of success for certain kinds of students.

I am worried though that online learning in California has ‘succeeded’ because of the massive cuts to campus-based education. It is better than nothing, but online learning deserves to be considered in its own right, not as a cheaper alternative to campus-based education. Online learning is not a panacea. Different students have different needs, and a successful public post secondary education system should cater to all needs. In the meantime, this is one of the most useful studies on online completion rates.

 

 

 

A balanced research report on the hopes and realities of MOOCs

Listen with webReader

Columbia MOOCs 2

Hollands, F. and Tirthali, D. (2014) MOOCs: Expectations and Reality New York: Columbia University Teachers’ College, Center for Benefit-Cost Studies of Education, 211 pp

We are now beginning to see a number of new research publications on MOOCs. The journal Distance Education will be publishing a series of research articles on MOOCs in June, but now Hollands and Tirthali have produced a comprehensive research analysis of MOOCs.

What the study is about

We have been watching for evidence that MOOCs are cost-effective in producing desirable educational outcomes compared to face-to-face experiences or other online interventions. While the MOOC phenomenon is not mature enough to afford conclusions on the question of long-term cost-effectiveness, this study serves as an exploration of the goals of institutions creating or adopting MOOCs and how these institutions define effectiveness of their MOOC initiatives. We assess the current evidence regarding whether and how these goals are being achieved and at what cost, and we review expectations regarding the role of MOOCs in education over the next five years. 

The authors used interviews with over 80 individuals covering 62 institutions ‘active in the MOOCspace’, cost analysis, and analysis of other research on MOOCs to support their findings. They identified six goals from the 29 institutions in the study that offered MOOCs, with following analysis of success or otherwise in accomplishing such goals:

1. Extending reach (65% 0f the 29 institutions)

Data from MOOC platforms indicate that MOOCs are providing educational opportunities to millions of individuals across the world. However, most MOOC participants are already well-educated and employed, and only a small fraction of them fully engages with the courses. Overall, the evidence suggests that MOOCs are currently falling far short of “democratizing” education and may, for now, be doing more to increase gaps in access to education than to diminish them. 

2. Building and maintaining brand (41%)

While many institutions have received significant media attention as a result of their MOOC activities, isolating and measuring impact of any new initiative on brand is a difficult exercise. Most institutions are only just beginning to think about how to capture and quantify branding-related benefits.

3. Reducing costs or increasing revenues (38%)

….revenue streams for MOOCs are slowly materializing but we do not expect the costs of MOOC production to fall significantly given the highly labor-intensive nature of the process. While these costs may be amortized across multiple uses and multiple years, they will still be additive costs to the institutions creating MOOCs. Free, non-credit bearing MOOCs are likely to remain available only from the wealthiest institutions that can subsidize the costs from other sources of funds. For most institutions, ongoing participation in the current MOOC experimentation will be unaffordable unless they can offer credentials of economic value to attract fee-paying participants, or can use MOOCs to replace traditional offerings more efficiently, most likely by reducing expensive personnel. 

4. Improving educational outcomes (38%)

for the most part, actual impact on educational outcomes has not been documented in any rigorous fashion. Consequently, in most cases, it is unclear whether the goal of improving educational outcomes has been achieved . However, there were two exceptions, providing evidence of improvement in student performance as a result of adopting MOOC strategies in on-campus courses

5. Innovation in teaching and learning (38%)

It is abundantly clear that MOOCs have prompted many institutions and faculty members to engage in new educational activities. The strategies employed online such as frequent assessments and short lectures interspersed with questions are being taken back on-campus. It is less clear what has been gained by these new initiatives because the value of innovation is hard to measure unless it can be tied to a further, more tangible objective. We …. conclude that most institutions are not yet making any rigorous attempt to assess whether MOOCs are more or less effective than other strategies to achieve these goals. 

6. Research on teaching and learning (28%)

A great deal of effort is being expended on trying to improve participant engagement and completion of MOOCs and less effort on determining whether participants actually gain skills or knowledge from the courses ….While the potential for MOOCs to contribute significantly to the development of personalized and adaptive learning is high, the reality is far from being achieved. 

Cost analysis

The report investigates the costs of developing MOOCs compared to those for credit-based online courses, but found wide variations and lack of reliable data.

Conclusions from the report

The authors came to the following conclusions:

1. there is no doubt that online and hybrid learning is here to stay and that MOOCs have catalyzed a shift in stance by some of the most strongly branded institutions in the United States and abroad.

2. MOOCs could potentially affect higher education in more revolutionary ways by:

  • offering participants credentials of economic value

  • catalyzing the development of true adaptive learning experiences

However, either of these developments face substantial barriers and will require major changes in the status quo.

My comments on the report

First this is an excellent, comprehensive and thoughtful analysis of the expectations and realities of MOOCs. It is balanced, but where necessary critical of the unjustified claims often made about MOOCs. This report should be required reading for anyone contemplating offering MOOCs.

Different people will take away different conclusions from this report, as one would expect from a balanced study. From my perspective, though, it has done little to change my views about MOOCs. MOOC providers to date have made little effort to identify the actual learning that takes place. It seems to be enough for many MOOC proponents to just offer a course, on the assumption that if people participate they will learn.

Nevertheless, MOOCs are evolving. Some of the best practices that have been used in credit-based online courses are now being gradually adopted as more MOOC players enter the market with experience of credit-based online learning. MOOCs will eventually occupy a small but important niche as an alternative form of non-formal, continuing and open education. They have proved valuable in making online learning more acceptable within traditional institutions that have resisted online learning previously. But no-one should fear them as a threat to credit-based education, either campus-based or online.

Research from the Michigan Virtual University on a connectivist MOOC

Listen with webReader

 MVU MOOC report

Ferdig, R. et al. (2014) Findings and reflections from the ‘K-12 Teaching in the 21st Century’ MOOC Lansing MI: Michigan Virtual Learning Research Institute

We are now beginning to get some in-depth research or evaluations of MOOCs. This one is from a team at Kent State University that developed a five week ‘connectivist’ MOOC aimed principally at three distinct audiences: high school students interested in becoming teachers, preservice teachers, and inservice teachers in the K-12 system.

I provide here a very brief summary of the report (as always, you should read the report for yourself if my summary gets you interested). Italics are direct quotes from the report.

Goal of the MOOC

How can we get teachers to think more deeply about reinventing education?

MOOC design

facilitators take on the role of connecting people around an idea for the purpose of bettering our understanding of the
idea. A connectivist-based MOOC draws on the extensive number of participants as well as the existing open repository of content to develop an experience. Participants are both teachers and learners in a process – not a product.

The course was designed around four principles often associated with teaching in the 21st century: connected learning, personalization, collaboration, and reflection.

Core technology

Coursesites by Blackboard provided the basic platform for content and discussion, supplemented by the use of participants’ social media networks and technologies. In addition participants were asked to create an ‘artifact’ to represent their learning.

Use of partners/co-facilitators

Kent State provided core facilitators for the MOOC, but they also invited other co-facilitators from schools, colleges and universities both in Michigan and from several other states.

Qualifications

Badges and continuing education units were given for successful participation.

Main results

Participants (data at time of enrollment, i.e. all participants)

Start of course: 673; end of course: 848; mainly from Michigan and surrounding states, although 12 were international

School teachers: 42%; k-12 students: 23%; post-secondary students: 16%; 19% other (inc. school administrators, university faculty); 80% female.

Participants’ response to the MOOC (168 participants who completed a post-course survey)

Most participants who responded enjoyed the MOOC, with in-service teachers enjoying it the most. Th main criticism (especially from the k-12 students) was the amount of work involved in following the MOOC.

Very active participation in the online discussion forums (within the Coursesites LMS)

There were over 6,000 actual posts (comments) and over 65,000 ‘hits’/looks over a five week period, from just over 300 of the participants – but almost to-thirds did not participate at all.

Types of participation

Lurkers (i.e. did not participate in LMS discussion forums – they may have participated through social media): 63%. There were accounts created in Facebook, Twitter, Delicious and blogs related to the course which indicated active social media connections both for registered participants and with those who had not registered for the course but were interested. However, these numbers were relatively small, and hard to measure.

Passive participation was defined as doing the minimum amount of work required to complete the course. Some of the passive participants were K-12 students forced to complete the MOOC for a class requirement.

There were also preservice teachers and inservice teachers who could be described as passive participants. These participants often completed the course; however, much like the high school students, their posts were limited to one or two sentences per posts. Their comments were also superficial, for example, “Nice job” or “I like what you did.”

Active participants participated in four ways:

  • informing personal practice
  • sharing the MOOC with their communities
  • leadership within the MOOC community
  • critical colleagues

The authors’ main conclusions

The seeking and sharing of digital media highlights that people want to form and engage in communities, and the growing interest in MOOCs shows this is true of educational communities as well….

Learning takes place in communities; depending on the implementation, technology has the capability to create and sustain the communities’ learning and practice….. Evidence in this report suggests that such activities can lead to positive outcomes, particularly as they relate to getting teachers to think more deeply about teaching and learning in the 21st century.

My comments

Even though (or perhaps because) this is a self-evaluation, this is a very useful report. I was fascinated for instance that this course ended with more participants than when it started, due to the ‘publicity’ of social media connections during the course itself.  It was interesting too that some of the participants in this MOOC were not necessarily willing participants – being forced to participate as part of a formal credit program. This seems to me to go against the whole purpose of a connectivist MOOC.

More importantly for me, the report highlights some of the ways research can be conducted on MOOCs and also some of the challenges. The study identifies the importance, from a research perspective, of having some kind of platform that can gather student data and track student behaviour, such as levels or types of participation. However, given the importance of social media for connectivist MOOCs, some way of accurately tracking related social media activity is critical. It seems to me that this is a problem that appropriate software could solve (further development of gRRShopper?), although privacy issues would need to be addressed as well. (Perhaps the spy agencies can help here – just joking!)

I agree completely with the authors when they write:

Researchers have already provided ample evidence that asking if a technology works is the wrong question. A more appropriate question is: under what conditions do certain types of MOOCs work?

Another even more pertinent question is: What prior research into credit-based online learning applies – and what does not apply – to different kinds of MOOCs? This might save a lot of time re-inventing the wheel, particularly for xMOOCs. I am getting sick of hearing from research on xMOOCs that immediate feedback helps retention – we have known that for nearly 100 years. We do need though for instance to assess the importance and most useful roles, if any, of instructors/facilitators/subject matter experts in MOOCs, and whether MOOCs can succeed with reduced ‘expert’ participation. This report suggests almost the opposite – connectivist MOOCs work best with a wide range of facilitators – but what are the hidden costs of this?

Finally, I also agree with the authors that completion rates are not the best measure of success for MOOCs. This MOOC does seem to have raised some interesting questions for participants. I’m just curious about their answers. Despite the very good work done by the instructors/researchers of this MOOC, I am still left with the question: what did the participants actually learn from this MOOC? For instance, what would an analysis of the student ‘artifacts’ have told us about their learning? Unless we try to answers questions about what actual learning took place then it will remain difficult if not impossible to measure the true value of different kinds of MOOC, and I think that would be a pity.

In the meantime, this report is definitely recommended reading for anyone interested in doing research on or evaluating MOOCs.

Contact North on How to Design an Innovative Course

Listen with webReader
Image © University of Ontario Institute of Technology, 2014

Image © University of Ontario Institute of Technology, 2014

Anon (2014) How to design an innovative course, Sudbury.Thunder Bay ON: Contact North

As reported in Contact North’s Online Learning News:

There is a lot of pressure these days on faculty and instructors to be ‘innovative’ in their teaching. But exactly what does being innovative mean? How do you go about designing and implementing an innovative course? What is the problem you are trying to solve? Will technology help? Learn a series of steps that can help you approach innovative teaching in a systematic way.

Comment

I found this article quite interesting. No author is given but they must be pretty smart to get this topic down to about 1,000 words….