April 19, 2015

The strengths and weaknesses of MOOCs: Part 2: learning and assessment

Listen with webReader
Remote exam proctoring

Remote exam proctoring

The writing of my book, Teaching in a Digital Age, has been interrupted for nearly two weeks by my trip to England for the EDEN Research Workshop. As part of the first draft of the book, I have already published three posts on MOOCs:

In this post, I ask (and try to answer) what do participants learn from MOOCs, and I also evaluate their assessment methods.

What do students learn in MOOCs?

This is a much more difficult question to answer, because so little of the research to date (2014) has tried to answer this question. (One reason, as we shall see, is that assessment of learning in MOOCs remains a major challenge). There are at least two kinds of study: quantitative studies that seek to quantify learning gains; and qualitative studies that describe the experience of learners within MOOCs, which indirectly provide some insight into what they have learned.

At the time of writing, the most well conducted study of learning in MOOCs has been by Colvin et al. (2014), who investigated ‘conceptual learning’ in an MIT Introductory Physics MOOC. They compared learner performance not only between different sub-categories of learners within the MOOC, such as those with no physics or math background with those such as physic teachers who had considerable prior knowledge, but also with on-campus students taking the same curriculum in a traditional campus teaching format. In essence, the study found no significant differences in learning gains between or within the two types of teaching, but it should be noted that the on-campus students were students who had failed an earlier version of the course and were retaking it.

This research is a classic example of the no significant difference in comparative studies in educational technology; other variables, such as differences in the types of students, were as important as the mode of delivery. Also, this MOOC design represents a behaviourist-cognitivist approach to learning that places heavy emphasis on correct answers to conceptual questions. It doesn’t attempt to develop the skills needed in a digital age as identified in Chapter 1.

There have been far more studies of the experience of learners within MOOCs, particularly focusing on the discussions within MOOCs (see for instance, Kop, 2011). In general (although there are exceptions), discussions are unmonitored, and it is left to participants to make connections and respond to other students comments.

However, there are some strong criticisms of the effectiveness of the discussion element of MOOCs for developing the high-level conceptual analysis required for academic learning. To develop deep, conceptual learning, there is a need in most cases for intervention by a subject expert, to clarify misunderstandings or misconceptions, to provide accurate feedback,  to ensure that the criteria for academic learning, such as use of evidence, clarity of argument, etc., are being met, and to ensure the necessary input and guidance to seek deeper understanding (see Harasim, 2013).

Furthermore, the more massive the course, the more likely participants are to feel ‘overload, anxiety and a sense of loss’, if there is not some instructor intervention or structure imposed (Knox, 2014). Firmin et al. (2014) have shown that when there is some form of instructor ‘encouragement and support of student effort and engagement’, results improve for all participants in MOOCs. Without a structured role for subject experts, participants are faced with a wide variety of quality in terms of comments and feedback from other participants. There is again a great deal of research on the conditions necessary for the successful conduct of collaborative and co-operative group learning (see for instance, Dillenbourg, 1999, Lave and Wenger, 1991), and these findings certainly have not been generally applied to the management of MOOC discussions to date.

One counter argument is that at least cMOOCs develop a new form of learning based on networking and collaboration that is essentially different from academic learning, and MOOCs are thus more appropriate to the needs of learners in a digital age. Adult participants in particular, it is claimed by Downes and Siemens, have the ability to self-manage the development of high level conceptual learning.  MOOCs are ‘demand’ driven, meeting the interests of individual students who seek out others with similar interests and the necessary expertise to support them in their learning, and for many this interest may well not include the need for deep, conceptual learning but more likely the appropriate applications of prior knowledge in new or specific contexts. MOOCs do appear to work best for those who already have a high level of education and therefore bring many of the conceptual skills developed in formal education with them when they join a MOOC, and therefore contribute to helping those who come without such skills.

Over time, as more experience is gained, MOOCs are likely to incorporate and adapt some of the findings from research on smaller group work for much larger numbers. For instance, some MOOCs are using ‘volunteer’ or community tutors (Dillenbourg, 2014).The US State Department has organized MOOC camps through US missions and consulates abroad to mentor MOOC participants. The camps include Fulbright scholars and embassy staff who lead discussions on content and topics for MOOC participants in countries abroad (Haynie, 2014). Some MOOC providers, such as the University of British Columbia, pay a small cohort of academic assistants to monitor and contribute to the MOOC discussion forums (Engle, 2014). Engle reported that the use of academic assistants, as well as limited but effective interventions from the instructors themselves, made the UBC MOOCs more interactive and engaging. However, paying for people to monitor and support MOOCs will of course increase the cost to providers. Consequently, MOOCs are likely to develop new automated ways to manage discussion effectively in very large groups. The University of Edinburgh is experimenting with automated ‘teacherbots’ that crawl through online discussion forums and direct predetermined comments to students identified as needing help or encouragement (Bayne, 2014).

These results and approaches are consistent with prior research on the importance of instructor presence for successful for-credit online learning. In the meantime, though, there is much work still to be done if MOOCs are to provide the support and structure needed to ensure deep, conceptual learning where this does not already exist in students. The development of the skills needed in a digital age is likely to be an even greater challenge when dealing with massive numbers. However, we need much more research into what participants actually learn in MOOCs and under what conditions before any firm conclusions can be drawn.

Assessment

Assessment of the massive numbers of participants in MOOCs has proved to be a major challenge. It is a complex topic that can be dealt with only briefly here. However, Chapter 5.8 provides a general analysis of different types of assessment, and Suen (2014) provides a comprehensive and balanced overview of the way assessment has been used in MOOCs to date. This section draws heavily on Suen’s paper.

Computer marked assignments

Assessment to date in MOOCs has been primarily of two kinds. The first is based on quantitative multiple-choice tests, or response boxes where formulae or ‘correct code’ can be entered and automatically checked. Usually participants are given immediate automated feedback on their answers, ranging from simple right or wrong answers to more complex responses depending on the type of response they have checked, but in all cases, the process is usually fully automated.

For straight testing of facts, principles, formulae, equations and other forms of conceptual learning where there are clear, correct answers, this works well. In fact, multiple choice computer marked assignments were used by the UK Open University as long ago as the 1970s, although the means to give immediate online feedback were not available then. However, this method of assessment is limited for testing deep or ‘transformative’ learning, and particularly weak for assessing the intellectual skills needed in a digital age, such as creative or original thinking.

Peer review

The second type of assessment that has been tried in MOOCs has been peer assessment, where participants assess each other’s work. Peer assessment is not new. It has been successfully used for formative assessment in traditional classrooms and in some online teaching for credit (Falchikov and Goldfinch, 2000; van Zundert et al., 2010). More importantly, peer assessment is seen as a powerful way to improve deep understanding and knowledge through the rating process, and at the same time, it can be useful for developing some of the skills needed in a digital age, such as critical thinking, for those participants assessing the work of others.

However, a key feature of the successful use of peer assessment has been the close involvement of an instructor or teacher, in providing benchmarks, rubrics or criteria  for assessment, and for monitoring and adjusting peer assessments to ensure consistency and a match with the benchmarks set by the instructor. Although an instructor can provide the benchmarks and rubrics in MOOCs, close monitoring of the multiple peer assessments is difficult if not impossible with the very large numbers of participants in MOOCs. As a result, MOOC participants often become incensed at being randomly assessed by other participants who may not and often do not have the knowledge or ability to give a ‘fair’ or accurate assessment of a participant’s work.

Various attempts to get round the limitations of peer assessment in MOOCs have been tried such as calibrated peer reviews, based on averaging all the peer ratings, and Bayesian post hoc stabilization, but although these statistical techniques reduce the error (or spread) of peer review somewhat they still do not remove the problems of systematic errors of judgement in raters due to misconceptions. This is particularly a problem where a majority of participants fail to understand key concepts in a MOOC, in which case peer assessment becomes the blind leading the blind.

Automated essay scoring

This is another area where there have been attempts to automate scoring. Although such methods are increasingly sophisticated they are currently limited in terms of accurate assessment to measuring primarily technical writing skills, such as grammar, spelling and sentence construction. Once again they do not measure accurately essays where higher level intellectual skills are demonstrated.

Badges and certificates

Particularly in xMOOCs, participants may be awarded a certificate or a ‘badge’ for successful completion of the MOOC, based on a final test (usually computer-marked) which measures the level of learning in a course. The American Council on Education (ACE), which represents the presidents of U.S. accredited, degree-granting institutions, recommended offering credit for five courses on the Coursera MOOC platform. However, according to the person responsible for the review process:

what the ACE accreditation does is merely accredit courses from institutions that are already accredited. The review process doesn’t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.’ (Book, 2013)

Indeed, most of the institutions offering MOOCs will not accept their own certificates for admission or credit within their own, campus-based programs. Probably nothing says more about the confidence in the quality of the assessment than this failure of MOOC providers to recognize their own teaching.

The intent behind assessment

To evaluate assessment in MOOCs requires an examination of the intent behind assessment. As identified earlier in another chapter of my book, there are many different purposes behind assessment. Peer assessment and immediate feedback on computer-marked tests can be extremely valuable for formative assessment, enabling participants to see what they have understood and to help develop further their understanding of key concepts. In cMOOCs, as Suen points out, learning is measured as the communication that takes place between MOOC participants, resulting in crowdsourced validation of knowledge – it’s what the sum of all the participants come to believe to be true as a result of participating in the MOOC, so formal assessment is unnecessary. However, what is learned in this way is not necessarily academically validated knowledge, which to be fair, is not the concern of cMOOC proponents such as Stephen Downes.

Academic assessment is a form of currency, related not only to measuring student achievement but also affecting student mobility (e.g. entrance to grad school) and perhaps more importantly employment opportunities and promotion. From a learner’s perspective, the validity of the currency – the recognition and transferability of the qualification – is essential. To date, MOOCs have been unable to demonstrate that they are able to assess accurately the learning achievements of participants beyond comprehension and knowledge of ideas, principles and processes (recognizing that there is some value in this alone). What MOOCs have not been able to demonstrate is that they can either develop or assess deep understanding or the intellectual skills required in a digital age. Indeed, this may not be possible within the constraints of massiveness, which is their major distinguishing feature from other forms of online learning, although the lack of valid methods of assessment will not stop computer scientists from trying to find ways to analyze participant online behaviour to show that such learning is taking place.

Up next

I hope the next post will be my last on this chapter on MOOCs. It will cover the following topics:

  • the cost of MOOCs and economies of scale
  • branding
  • the political, economic and social factors that explain the rise of MOOCs.

Over to you

As regular readers know, this is my way of obtaining peer review for my open textbook (so clearly I am not against peer review in principle!). So if I have missed anything important on this topic, or have misrepresented people’s views, or you just plain disagree with what I’ve written, please let me know. In particular, I am hoping for comments on:

  • comprehensiveness of the sources used that address learning and assessment methods in MOOCs
  • arguments that should have been included, either as a strength or a weakness
  • errors of fact

Yes, I’m a glutton for punishment, but you need to be a masochist to publish openly on this topic.

References

Bayne, S. (2014) Teaching, Research and the More-than-Human in Digital Education Oxford UK: EDEN Research Workshop (url to come)

Book, P. (2103) ACE as Academic Credit Reviewer–Adjustment, Accommodation, and Acceptance WCET Learn, July 25

Colvin, K. et al. (2014) Learning an Introductory Physics MOOC: All Cohorts Learn Equally, Including On-Campus Class, IRRODL, Vol. 15, No. 4

Dillenbourg, P. (ed.) (1999) Collaborative-learning: Cognitive and Computational Approaches. Oxford: Elsevier

Dillenbourg, P. (2014) MOOCs: Two Years Later, Oxford UK: EDEN Research Workshop (url to come)

Engle, W. (2104) UBC MOOC Pilot: Design and Delivery Vancouver BC: University of British Columbia

Falchikov, N. and Goldfinch, J. (2000) Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks Review of Educational Research, Vol. 70, No. 3

Firmin, R. et al. (2014) Case study: using MOOCs for conventional college coursework Distance Education, Vol. 35, No. 2

Haynie, D. (2014). State Department hosts ‘MOOC Camp’ for online learners. US News,January 20

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Ho, A. et al. (2014) HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013 (HarvardX and MITx Working Paper No. 1), January 21

Knox, J. (2014) Digital culture clash: ‘massive’ education in the e-Learning and Digital Cultures Distance Education, Vol. 35, No. 2

Kop, R. (2011) The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course International Review of Research into Open and Distance Learning, Vol. 12, No. 3

Lave, J. and Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press

Milligan, C., Littlejohn, A. and Margaryan, A. (2013) Patterns of engagement in connectivist MOOCs, Merlot Journal of Online Learning and Teaching, Vol. 9, No. 2

Suen, H. (2104) Peer assessment for massive open online courses (MOOCs) International Review of Research into Open and Distance Learning, Vol. 15, No. 3

van Zundert, M., Sluijsmans, D., van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20, 270-279

Review of ‘Online Distance Education: Towards a Research Agenda.’

Listen with webReader
Drop-out: the elephant in the DE room that no-one wants to talk about

Drop-out: the elephant in the DE room that no-one wants to talk about

Zawacki-Richter, O. and Anderson, T. (eds.) (2014) Online Distance Education: Towards a Research Agenda Athabasca AB: AU Press, pp. 508

It is somewhat daunting to review a book of over 500 pages of research on any topic. I doubt if few other than the editors are likely to read this book from cover to cover. It is more likely to be kept on one’s bookshelf (if these still exist in a digital age) for reference whenever needed. Nevertheless, this is an important work that anyone working in online learning needs to be aware of, so I will do my best to cover it as comprehensively as I can.

Structure of the book

The book is a collection of about 20 chapters by a variety of different authors (more on the choice of authors later). Based on a Delphi study and analysis of ‘key research journals’ in the field, the editors have organized the topic into three sections, with a set of chapters on each sub-section, as follows:

1. Macro-level research: distance education systems and theories

  • access, equity and ethics
  • globalization and cross-cultural issues
  • distance teaching systems and institutions
  • theories and models
  • research methods and knowledge transfer

2. Meso-level research: management, organization and technology

  • management and organization
  • costs and benefits
  • educational technology
  • innovation and change
  • professional development and faculty support
  • learner support services
  • quality assurance

3. Micro-level: teaching and learning in distance education

  • instructional/learning design
  • interaction and communication
  • learner characteristics.

In addition, there is a very useful preface from Otto Peters, an introductory chapter by the editors where they justify their structural organization of research, and a short conclusion that calls for a systematic research agenda in online distance education research.

More importantly, perhaps, Terry Anderson and Olaf Zawacki-Richter demonstrate empirically that research in this field has been skewed towards micro-level research (about half of all publications).  Interestingly, and somewhat surprisingly given its importance, costs and benefits of online distance education is the least researched area.

What I liked

It is somewhat invidious to pick out particular chapters, because different people will have different interests from such a wide-ranging list of topics. I have tended to choose those that I found were new and/or particularly enlightening for me, but other readers’ choices will be different. However, by selecting a few excellent chapters, I hope to give some idea of the quality of the book.

1. The structuring/organization of research

Anderson and Zawacki-Richter have done an excellent job in providing a structural framework for research in this field. This will be useful both for those teaching about online and distance education but in particular for potential Ph.D. students wondering what to study. This book will provide an essential starting point.

2. Summary of the issues in each area of research

Again, the editors have done an excellent job in their introductory chapter in summarizing the content of each of the chapters that follows, and in so doing pulling out the key themes and issues within each area of research. This alone makes the book worthwhile.

3. Globalization, Culture and Online Distance Education

Charlotte (Lani) Gunawardena of the University of New Mexico has written the most comprehensive and deep analysis of this issue that I have seen, and it is an area in which I have a great deal of interest, since most of the online teaching I have done has been with students from around the world and sometimes multi-lingual.

After a general discussion of the issue of globalization and education, she reviews research in the following areas:

  • diverse educational expectations
  • learners and preferred ways of learning
  • socio-cultural environment and online interaction
  • help-seeking behaviours
  • silence
  • language learning
  • researching culture and online distance learning

This chapter should be required reading for anyone contemplating teaching online.

4. Quality assurance in Online Distance Education

I picked this chapter by Colin Latchem because he is so deeply expert in this field that he is able to make what can be a numbingly boring but immensely important topic a fun read, while at the same time ending with some critical questions about quality assurance. In particular Latchem looks at QA from the following perspectives:

  • definitions of quality
  • accreditation
  • online distance education vs campus-based teaching
  • quality standards
  • transnational online distance education
  • open educational resources
  • costs of QA
  • is online distance education yet good enough?
  • an outcomes approach to QA.

This chapter definitely showcases a master at the top of his game.

5. The elephant in the room: student drop-out

This is a wonderfully funny but ultimately serious argument between Ormond Simpson and Alan Woodley about the elephant in the distance education room that no-one wants to mention. Here they start poking the elephant with some sticks (which they note is not likely to be a career-enhancing move.) The basic argument is that institutions should and could do more to reduce drop-out/increase course completion. This chapter also stunned me with providing hard data about really low completion rates for most open university students. I couldn’t help comparing these with the high completion rates for online credit courses at dual-mode (campus-based) institutions, at least in Canada (which of course are not ‘open’ institutions in that students must have good high school qualifications.)

Woodley’s solution to reducing drop-out is quite interesting (and later well argued):

  • make it harder to get in
  • make it harder to get out

In both cases, really practical and not too costly solutions are offered that nevertheless are consistent with open access and high quality teaching.

In summary

The book contains a number of really good chapters that lay out the issues in researching online distance education.

What I disliked

I have to say that I groaned when I first saw the list of contributors. The same old, same old list of distance education experts with a heavy bias towards open universities. Sure, they are nearly all well-seasoned experts, and there’s nothing wrong with that per se (after all, I see myself as one of them.)

But where are the young researchers here, and especially the researchers in open educational resources, MOOCs, social media applications in online learning, and above all researchers from the many campus-based universities now mainstreaming online learning? There is almost nothing in the book about research into blended learning, and flipped classrooms are not even mentioned. OK, the book is about online distance learning but the barriers or distinctions are coming down with a vengeance. This book will never reach those who most need it, the many campus-based instructors now venturing for the first time into online learning in one way or another. They don’t see themselves as primarily distance educators.

And a few of the articles were more like lessons in history than an up-to-date review of research in the field. Readers of this blog will know that I strongly value the history of educational technology and distance learning. But these lessons need to be embedded in the here and now. In particular, the lessons need to be spelled out. It is not enough to know that Stanford University researchers as long ago as 1974 were researching the costs and benefits of educational broadcasting in developing countries, but what lessons does this have for some of the outrageous claims being made about MOOCs? A great deal in fact, but this needs explaining in the context of MOOCs today.

Also the book is solely focused on post-secondary university education. Where is the research on online distance education in the k-12/school sector or the two-year college/vocational sector? Maybe they are topics for other books, but this is where the real gap exists in research publications in online learning.

Lastly, although the book is reasonably priced for its size (C$40), and is available as an e-text as well as the fully printed version, what a pity it is not an open textbook that could then be up-dated and crowd-sourced over time.

Conclusion

This is essential reading for anyone who wants to take a professional, evidence-based approach to online learning (distance or otherwise). It will be particularly valuable for students wanting to do research in this area. The editors have done an incredibly good job of presenting a hugely diverse and scattered area in a clear and structured manner. Many of the chapters are gems of insight and knowledge in the field.

However, we have a huge challenge of knowledge transfer in this field. Repeatedly authors in the book lamented that many of the new entrants to online learning are woefully ignorant of the research previously done in this field. We need a better way to disseminate this research than a 500 page printed text that only those already expert in the field are likely to access. On the other hand, the book does provide a strong foundation from which to find better ways to disseminate this knowledge. Knowledge dissemination in a digital world then is where the research agenda for online learning needs to focus.

 

The role of communities of practice in a digital age

Listen with webReader
Bank of America's Vital Voices progam links women executives of small and medium sized enterprises  Image: © Belfast Telegraph, 2014

Bank of America’s Vital Voices progam links women executives of small and medium sized enterprises from around the world
Image: © Belfast Telegraph, 2014

The story so far

I have published the first five chapters of my open textbook, ‘Teaching in a Digital Age‘.  I am now working on Chapter 6, ‘Models for Designing Teaching and Learning.’

In my last three posts I discussed respectively the appropriateness for a digital age of the classroom model, the ADDIE model, and the competency-based learning model. In this post, I explore the learning model based on communities of practice.

The theories behind communities of practice

The design of teaching often integrates different theories of learning. Communities of practice are one of the ways in which experiential learning, social constructivism, and connectivism can be combined, illustrating the limitations of trying to rigidly classify learning theories. Practice tends to be more complex.

What are communities of practice?

Definition: 

Communities of practice are groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly.

Wenger, 2014

The basic premise behind communities of practice is simple: we all learn in everyday life from the communities in which we find ourselves. Communities of practice are everywhere. Nearly everyone belongs to some community of practice, whether it is through our working colleagues or associates, our profession or trade, or our leisure interests, such as a book club. Wenger (2000) argues that a community of practice is different from a community of interest or a geographical community in that it involves a shared practice: ways of doing things that are shared to some significant extent among members.

Wenger argues that there are three crucial characteristics of a community of practice:

  • domain: a common interest that connects and holds together the community
  • community: a community is bound by the shared activities they pursue (for example, meetings, discussions) around their common domain
  • practice: members of a community of practice are practitioners; what they do informs their participation in the community; and what they learn from the community affects what they do.

Wenger (2000) has argued that although individuals learn through participation in a community of practice, more important is the generation of newer or deeper levels of knowledge through the sum of the group activity. If the community of practice is centered around business processes, for instance, this can be of considerable benefit to an organization. Smith (2003) notes that:

…communities of practice affect performance..[This] is important in part because of their potential to overcome the inherent problems of a slow-moving traditional hierarchy in a fast-moving virtual economy. Communities also appear to be an effective way for organizations to handle unstructured problems and to share knowledge outside of the traditional structural boundaries. In addition, the community concept is acknowledged to be a means of developing and maintaining long-term organizational memory.

Brown and Duguid (2000) describe a community of practice developed around the Xerox customer service representatives who repaired the machines in the field. The Xerox reps began exchanging tips and tricks over informal meetings at breakfast or lunch and eventually Xerox saw the value of these interactions and created the Eureka project to allow these interactions to be shared across the global network of representatives. The Eureka database has been estimated to have saved the corporation $100 million. Companies such as Google and Apple are encouraging communities of practice through the sharing of knowledge across their many specialist staff.

Technology provides a wide range of tools that can support communities of practice, as indicated by Wenger (2010) in the diagram below:

Image: © Etienne Wenger, 2010

Designing effective communities of practice

Most communities of practice have no formal design and tend to be self-organising systems. They have a natural life cycle, and come to an end when they no longer serve the needs of the community. However, there is now a body of theory and research that has identified actions that can help sustain and improve the effectiveness of communities of practice.

 Wenger, McDermott and Snyder (2002) have identified seven key design principles for creating effective and self-sustaining communities of practice, related specifically to the management of the community, although the ultimate success of a community of practice will be determined by the activities of the members of the community themselves. Designers of a community of practice need to:

  1. Design for evolution: ensuring that the community can evolve and shift in focus to meet the interests of the participants without moving too far from the common domain of interest
  2. Open a dialogue between inside and outside perspectives: encourage the introduction and discussion of new perspectives that come or are brought in from outside the community of practice
  3. Encourage and accept different levels of participation, The strength of participation varies from participant to participant. The ‘core’ (most active members) are those who participate regularly. There are others who follow the discussions or activities but do not take a leading role in making active contributions. Then there are those (likely the majority) who are on the periphery of the community but may become more active participants if the activities or discussions start to engage them more fully. All these levels of participation need to be accepted and encouraged within the community.
  4. Develop both public and private community spaces: communities of practice are strengthened if they encourage individual or group activities that are more personal or private as well as the more public general discussions; for instance, individuals may decide to blog about their activities, or in a larger online community of practice a small group that live or work close together may also decide to meet informally on a face-to-face basis
  5. Focus on value. Attempts should be made explicitly to identify, through feedback and discussion, the contributions that the community most values, then focus the discussion and activities around these issues.
  6. Combine familiarity and excitement, by focusing both on shared, common concerns and perspectives, but also by introducing radical or challenging perspectives for discussion or action
  7. Create a rhythm for the community: there needs to be a regular schedule of activities or focal points that bring participants together on a regular basis, within the constraints of participants’ time and interests.

Subsequent research has identified a number of critical factors that influence the effectiveness of participants in communities of practice, These include being:

  • aware of social presence: individuals need to feel comfortable in engaging socially with other professionals or ‘experts’ in the domain, and those with greater knowledge must be willing to share in a collegial manner that respects the views and knowledge of other participants (social presence is defined as the awareness of others in an interaction combined with an appreciation of the interpersonal aspects of that interaction.)
  • motivated to share information for the common good of the community
  • able and willing to collaborate.

EDUCAUSE has developed a step-by-step guide for designing and cultivating communities of practice in higher education (Cambridge, Kaplan and Suter, 2005).

Lastly, research on other related sectors, such as collaborative learning or MOOCs, can inform the design and development of communities of practice. For instance, communities of practice need to balance between structure and chaos: too much structure and many participants are likely to feel constrained in what they need to discuss; too little structure and participants can quickly lose interest or become overwhelmed.

Many of the other findings about group and online behaviour, such as the need to respect others, observing online etiquette, and preventing certain individuals from dominating the discussion, are all likely to apply. However, because many communities of practice are by definition self-regulating, establishing rules of conduct and even more so enforcing them is really a responsibility of the participants themselves.

Learning through communities of practice in a digital age

Communities of practice are a powerful manifestation of informal learning. They generally evolve naturally to address commonly shared interests and problems. By their nature, they tend to exist outside formal educational organisations. Participants are not usually looking for formal qualifications, but to address issues in their life and to be better at what they do. Furthermore, communities of practice are not dependent on any particular medium; participants may meet face-to-face socially or at work, or they can participate in online or virtual communities of practice.

It should be noted that communities of practice can be very effective in a digital world, where the working context is volatile, complex, uncertain and ambiguous.  A large part of the lifelong learning market will become occupied by communities of practice and self-learning, through collaborative learning, sharing of knowledge and experience, and crowd-sourcing new ideas and development. Such informal learning provision will be particularly valuable for non-governmental or charitable organizations, such as the Red Cross, Greenpeace or UNICEF, or local government, looking for ways to engage communities in their areas of operation.

These communities of learners will be open and free, and hence will provide a competitive alternative to the high priced lifelong learning programs being offered by research universities. This will put pressure on universities and colleges to provide more flexible arrangements for recognition of informal learning, in order to hold on to their current monopoly of post-secondary accreditation.

One of the significant developments in recent years has been the use of massive open online courses (MOOCs) for developing online communities of practice. To date the focus of the majority of MOOCs from providers such as Coursera, Udacity and edX, has been on academic ‘courses’, on topics such as artificial intelligence or dinosaurs, which do have a widespread interest. However, these more instructionist MOOCs are not really developed as communities of practice, because  they use mainly a transmissive pedagogy, from experts to those considered less expert. Even though there may be massive numbers participating in online forums, they are not constructed to maximise the contributions from the participants (despite the fact that most MOOC participants already have high levels of education.). Indeed there is evidence that in really large instructionist MOOCs, participants feel overwhelmed by the magnitude and lack of structuring of the participant contributions (see for instance Knox, 2014).

In comparison, connectivist MOOCs are an ideal way to bring together specialists scattered around the world to focus on a common interest or domain. Connectivist MOOCs are much closer to being virtual communities of practice, in that they put much more emphasis on sharing knowledge between more or less equal participants. However, current connectivist MOOCs do not always incorporate what research indicates are best practices for developing communities of practice, and those wanting to establish a virtual community of practice at the moment need some kind of MOOC provider to get them started and give them access to the necessary MOOC software.

In the long run, MOOCs need to evolve to the point where it is possible for those with a common interest to easily create their own open, online communities of practice. As open source MOOC platforms evolve, it should become easier for people without computer science degrees to create and more importantly manage their own MOOCs, without having to go through a MOOC provider such as Coursera or edX. Also, there are other simpler tools, such as wikis, or more complex ones, such as virtual worlds, that may in the long run have more potential for virtual communities of practice created and organised by the participants themselves.

Although communities of practice are likely to become more rather than less important in a digital age, it is probably a mistake to think of them as a replacement for traditional forms of education. There is no single, ‘right’ approach to the design of teaching. Different groups have different needs. Communities of practice are more of an alternative for certain kinds of learners, such as lifelong learners, and are likely to work best when participants already have some domain knowledge and can contribute personally and in a constructive manner – which suggests the need for at least some form of prior general education or training for those participating in effective communities of practice.

In conclusion, it is clear is that in an increasingly volatile, uncertain, complex, and ambiguous world, and given the openness of the Internet, the social media tools now available, and the need for sharing of knowledge on a global scale, virtual communities of practice will become even more common and important. Smart educators and trainers will look to see how they can harness the strength of this design model, particularly for lifelong learning. However, merely lumping together large numbers of people with a common interest is unlikely to lead to effective learning.  Attention needs to be paid to those design principles that lead to effective communities of practice.

Over to you

Once again, I’m not an expert on communities of practice, so feedback on what I have written on this model of learning will be much appreciated. In particular:

  • Have a got it wrong? Are their important elements missing?
  • Is there good research on the design of communities of practice that I have missed?
  • Do you agree that they are NOT a replacement for other forms of education?
  • Can you really design a community of practice or do they just evolve naturally? If so, what conditions are needed for success other than those already discussed in this post?
  • How would you evaluate the success of a community of practice? What would you look for? How could this be identified or described?

I would love to hear from anyone who has attempted to create communities of practice and what design elements they would recommend.

References

Brown, J. and Duguid, Paul (2000). “Balancing act: How to capture knowledge without killing it”Harvard Business Review.

Cambridge, D., Kaplan, S. and Suter, V. (2005) Community of Practice Design Guide Louisville  CO: EDUCAUSE

Knox, J. (2004) Digital culture clash: “massive” education in the E-learning and Digital Cultures MOOC Distance Education, Vol. 35, No. 2

Wenger, E. (2000) Communities of Practice: Learning, Meaning and Identity Cambridge UK: Cambridge University Press

Wenger, E. (2014) Communities of practice: a brief introduction, accessed 26 September, 2014

Wenger, E, McDermott, R., and Snyder, W. (2002). Cultivating Communities of Practice (Hardcover). Harvard Business Press; 1 edition.

 

Is there a Canadian market for American online programs?

Listen with webReader
© Wikipedia, 2014

Image: © Wikipedia, 2014

Identifying the market

I got a phone call last week from an international consultancy company contracted on behalf of a major U.S. state university, wanting to know if I thought there was a Canadian market for U.S. online degrees. Since I didn’t get paid for my advice, I’m willing to share it with anyone who’s interested. (I do find it annoying that universities will pay large sums to general business consultants such as Price Waterhouse, KPMG, Accenture, etc., who then expect professionals in the field to do their work for them for free.)

So I will share with you the short answer I gave to the consultants, then the long one which only you have access to!

The short answer is yes, but only in niche markets. This means doing careful market research (not just ringing up a few experts), developing a strong business plan, keeping tuition fees relatively low compared with those at the top end of the market in the USA, and being nimble, in that the market will not last long if they are successful, because Canadian institutions will eventually try to take that market back.

Now for the long answer.

Why Canada is not an easy market for foreign online programs

I don’t have any figures, but there are certainly some Canadian students already taking online degrees from U.S. institutions (if you are one of them, I’d be interested in your experience.) Why would they do that, you might ask? Well,

  • if they want to get a job in the USA or work for an American company in Canada
  • if there is no Canadian university offering the program either face-to-face locally or online
  • because they think a big name U.S. university will do a better job than their local Canadian universities.

I’m not saying they are right to think that, but the whole question of comparative quality in post-secondary educations, either in Canada or the USA, is so fraught and subjective that potential students still have a very hard job making those kinds of decisions.

However, for that very reason, better the devil you know. In general, I would argue that most potential students in Canada will look to one of their local universities as their first choice, for several reasons:

  • Cost. Tuition fees are generally lower in Canada in absolute terms, but also Canadians may be eligible for grants and scholarships for study, which probably will not apply to online programs offered by a program from the USA (although again there are exceptions.)
  • Quality. Nearly all Canadian universities are publicly funded and accredited by the provincial government. There are of course differences in status, but the system is simple enough for most Canadians to be aware of these differences, and the variation between institutions in Canada is nowhere near as great as in the much larger and much more diverse US higher education system. In particular, distance and online education in the USA has been tarnished much more than in Canada by private, for profit diploma mills, making many Canadians suspicious then of any online programs offered from the USA
  • Access. In general, most Canadians can get access to the programs they need from one or more of their local universities. 51% of Canadians go on from high school to university, and 60% to some form of publicly-funded post-secondary education. There are already roughly one million course enrollments a year in online courses from Canadian post-secondary institutions, or roughly about 20-25 per cent of all course enrollments

So the Canadian market for post-secondary education in general is what marketers call ‘mature’ and even online learning, while not mature, is certainly not in its infancy in Canada. Many Canadian universities already offer a range of online programs, both for credit and for non-credit.

For instance, there are several online/distance MBAs (Athabasca University, Queen’s/Cornell, Laurentian, for instance) and UBC has been running several very successful online masters programs in education, creative writing and health sciences. This is a market where institutions can more than cover their costs within the current range of government-regulated tuition fees; indeed UBC has been able to add new research faculty as a result of some their online masters programs.

The Canadian Virtual University (CVU), which is a consortium of 11 universities from across Canada, lists over 2,000 online courses from its members, including a total of 57 online masters programs. There are 16 online masters programs in Ontario, Canada’s largest province (with 20 universities). Of the online masters currently available from Ontario institutions, one third are in education, one third in business or leadership, and one third in health or social work (see Ontario Online Learning Portal for Students). There are a few other universities in Canada that are not members of the CVU or in Ontario that also offer online programs, and the number grows each year. Thus there is already a significant Canadian university (and college) ‘presence’ in the online learning area.

Furthermore once a U.S. program starts attracting large numbers of Canadian students, this is likely to draw a response from the Canadian institutions. This happened in British Columbia in the 1990s, when a large number of teachers started taking distance education masters’ programs from Washington State institutions such as Gonzaga University, because there was little available in British Columbia. It took a few years but the B.C. institutions eventually responded and the number of teachers taking out of province online programs is now very much reduced.

Lastly there are more subtle cultural barriers. Canada is not so different from the USA as Canadians like to believe, but there are differences, and they are important. Our laws and government regulations for instance are different. We have different iconic references, different national symbols, a different racial mix. This means content does not always transfer as seamlessly as you would think for countries sharing a similar language and continent. This is not an insurmountable obstacle, but it does require a certain sensitivity to ‘Canadian-ness’ that is not always there in the USA. However, it is probably too costly to adapt existing materials for a specifically Canadian market.

So there are several reasons why some caution would be needed in marketing online programs from the USA for a Canadian market (or anywhere else, such as the U.K. or Australia).

Why there is still a market for foreign online programs in Canada

But Canadian universities should not be complacent. Canadian institutions have been slower than their counterparts in the USA in entering the online professional masters’ market. There are many gaps in the availability of post-graduate online programs in Canada, especially in the STEM (science, technology, engineering, math) areas.

For instance, in Ontario there are no online masters’ programs in food and agriculture, humanities (except Design from OCAD), social sciences, math, science, law or engineering. The situation would be similar or worse in the other provinces. In particular, despite the CVA, which covers only 11 of the 80 or so universities in Canada, there is no national portal for Canadian online courses, so potential students have a hard time looking for online programs offered out of province.

Secondly, should Harvard, Stanford or MIT start offering for-credit online programs, there would certainly be a large take-up in Canada of such programs. For the still prestigious, Tier 1 research state universities, such as the University of California, Arizona State University or Penn State, brand recognition in Canada would be less compelling if there are suitable Canadian alternatives, although for graduate programs, many Canadian students might still be interested.

Lastly, good marketing is likely to attract potential students in Canada who may well not be aware that similar programs are already available in Canada where they do exist. Even as a so-called expert in Canadian online learning, I was unaware of most of the programs listed in the CVA and Contact North portals. Canadian universities are still remarkably provincially focused when it comes to marketing online programming. Which brings me to my next point.

Why don’t Canadian institutions market online programs in the USA?

Well, some do, but not many, and there are good reasons for this.

First the U.S. accreditation system is byzantine and bizarre, and totally ill-adapted to the move to online, distance education, but without accreditation from one of the regional accreditation agencies in the U.S., any Canadian online program would not be recognized for financial assistance, credit transfer or by many employers. Indeed (although the federal regulations are currently under discussion) it may be illegal to offer programs to students within a state without approval from that state (and to get approval you may need a ‘physical presence.’) Despite the very large cost in getting regional accreditation, some specialist Canadian institutions, such as Athabasca University, have done so. I’d be interested to know though if they feel it was worth it, in terms of the numbers of U.S. students it has attracted. (Yes, you are correct: the U.S.A and Canada are partners in a free trade association called NAFTA, but it doesn’t apply to education – or forestry, for that matter.)

Second, the reverse is true regarding brand recognition and local preferences. Many U.S. citizens don’t even know where Canada is, let alone know whether the University of Waterloo is a bona fide institution and not a diploma mill. (I once shared an airport shuttle bus in Los Angeles with two American businessmen complaining they needed a passport to go to Toronto. ‘Anyone would think it was in another country,’ one said. I’m not making it up, and for British Columbian’s it may well be.)

So Canada, and Canadian institutions, do not have a strong enough brand image for most Americans, although some institutions, such as the University of Windsor, have done an excellent job in building partnership with institutions just across the border in Michigan and Detroit – but not, so far, in online learning.

Why these are the wrong questions

The real question should be (for both Canadian and U.S. online program developers): Is there an international market for online professional masters’ programs, to which the answer is a resounding ‘yes.’

It really doesn’t matter what country a student is in. If the program being offered meets a need, and the student can afford it, then they will take it, whether it comes from the USA, Canada, Australia, Spain, India or Brazil. Brand recognition helps. Canadian public institutions produce high quality, government accredited online programs. There is no distinction in the degree transcripts whether the program is offered at a distance or on campus. With its rather monolithic public higher education system, it would have an advantage if it could only market a Canadian rather than a local or provincial brand (but then it has the same problem in attracting international students to on-campus programs).

U.S. states trying to regulate whether students can take a program from out of state are like King Canute trying to stop the tide from coming in. All you can do is say ‘Caveat Emptor’ – buyer beware.

One way round the ‘not invented here’ attitude to out of state or out of country programs though is to develop a partnership with local institutions of a similar ‘status’ level. The University of British Columbia developed a very successful partnership with Tec de Monterrey in Mexico for a graduate program in educational technology. UBC developed most of the content, but the program was available in Spanish from Tec de Monterrey and in English from UBC. UBC’s offering of the program drew almost 30 per cent of its enrollments initially from English-speaking students from other countries all round the world. Similarly, Tec de Monterrey marketed its program in South America as well as in Mexico. Queen’s University in Canada has partnered with Cornell University in the USA to market a joint online MBA. These partnerships help to get round the accreditation issues.

Admission of international students to an online program can be more of a problem, as can issues of security in terms of exams and assessment, but these are not insuperable problems. If the institution already has an admission policy for on-campus international students this should apply just as well to online students from another country. Local proctoring (again using a partner institution) can help with examinations, but even better is to use authentic assessments where students have to apply their knowledge locally or within their own work or social context. This form of assessment is ideal for professional masters ‘ programs.

Furthermore, it helps when initially planning an online program to take into consideration that there is probably an international market (indeed, this helps with marketing the program even locally). This means choosing content carefully so it can work in a wide variety of international contexts, and to take advantage of the different perspectives from the international students.

Lastly, when developing a business model, the inclusion of international students can make the difference between loss and profit, because in well-designed online programs, the marginal cost of each additional student should more than cover the delivery costs. UBC found it did not need to charge a premium tuition fee for international students, because the business model depended on getting enough enrollments at the government-regulated tuition fee (for domestic students) to break even. The international students were an important contributor to breaking even without having to charge them a premium price. So pricing policy is also an important factor.

In conclusion

Yes, there is a market for international students in online programs, and there will be a market in Canada for specific professional masters offered by well-recognised and legitimate U.S. universities, provided they fill a gap not currently being met by Canadian institutions – and at the moment there are many such gaps.

However, it may be a mistake to focus on marketing to specific countries. Institutions will (even in Canada) respond relatively quickly to what they may feel as foreign incursions into their market. The world market (especially for programs in the English language) is large enough to attract more than enough high quality students to credit-based programs – just look at the response to MOOCs, which don’t offer formal qualifications. The trick is good marketing, making sure the materials take into consideration an international market and are not focused on specifically local or national issues, and having a good business plan. And partnership with foreign institutions would be a wise move. So while not easy, it is feasible, but if the reason is solely to make money, then it will, ironically, be a harder sell. Students will pick up on that remarkably quickly.

 

 

 

 

 

 

Another e-learning platform from Nigeria

Listen with webReader

Fora.co

Adepoju, P. (2014) Nigeria is ready for e-learning – Fora.co Humanipo, January 28

I wrote about tutor.ng in a previous post. Fora.co is another e-learning platform, working with ‘Africa’s leading Universities, organizations and governments to provide young Africans with affordable access to the best educational content online, offline and on mobile‘. It offers over 500 courses or course packs, consisting of:

pre-built bundles of relevant digital learning resources that can be used as teaching or training aids in the classroom. (Minimum 6 hours of professor lectures) 
+ Lab Exercises 
+ Textbooks & required readings 
+ Test banks (Minimum 100 questions) 
+ Presentation Slide templates for lectures

Iyinoluwa Aboyeji, chief executive officer at Fora.co, believes that in Nigeria, the technology is not the main barrier. The problem is lack of local content:

We are still light years behind others countries. Nigerian e-learning content is often badly designed and instructional design for online courses still seems foreign to our e-learning content landscape. This is one of the reasons we have had to focus on selling foreign courses because the local courses we saw were quite simply not up to snuff.

For students with difficult or costly access to the Internet, Fora provides learners with a flash drive that ‘synchronizes data from offline interactions and downloads new content from Fora.co to the flashdrive whenever internet access becomes available‘.

Flora markets both directly to institutions and also to individual students. Fora charges a fee per student that depends on the size of the institution and the kind of content bundle required. The lowest priced content bundle is $59.99/student (~N10,000/student).

However, at the moment its web site does not list the courses, the institutions that provide the materials, or the institutions that Fora is working with. This will come shortly; the materials however are properly sourced with the permission of each of the institutions whose materials are used.

Comment

Again, it will be interesting to see how this company develops, and whether the business model is successful. It is likely to work best with small, private institutions who can charge a premium fee thus generating a profit.

A major test will be if any African public universities partner, and whether courses will eventually be accredited in Nigeria.

Nevertheless I am sure we will see more attempts like this around Africa to build viable e-learning or online systems through the private sector.

Footnote

After I initially posted this, I discovered that this project had a Canadian origin, originally conceived at the University of Waterloo’s Velo City Garage and with connections with the MaRS Tech project: click here for much more information about the Fora operation. See also Iyinoluwa Aboyeji’s comments to this post below.