May 18, 2013

Follow on Twitter

You are here: Home / Archives for Teaching and learning / evaluating courses and programs

The role of information sciences in online learning: a review of IRRODL, Vol. 13, No. 5

January 3, 2013 By 3 Comments
Listen with webReader

An overview of the papers

IRRODL (the International Review of Research in Open and Distance Learning) has once again produced a fascinating themed edition, this time about the application of information science approaches to online learning. The issue has been promoted, reviewed, and edited by a skilled team of researchers led by editors Dr. Maiga Chang (Athabasca University, Canada), Dr. Rita Kuo (Knowledge Square Inc., Taiwan), Dr. Gene Loeb (Center for Technology and Mental Health of Elderly, USA), and Dr. Bolanle Olaniran (Texas Tech University, USA).

I provide at the end of this post a very brief summary of the papers, to give some indication of the range of topics. As Terry Anderson, the journal editor in chief, says: ‘The issue is a bit more techie than our usual offering’, and the articles certainly warrant careful reading, but are well worth it. I will provide here my personal reflections on what the articles, taken as a whole, suggest for future developments in online learning, although it should be pointed out that although all the articles are looking at computer-based approaches to issues in online learning, within them they reflect a wide variety of positions on the role of computers.

Computers and teachers

First I should lay out my inbuilt bias or prejudice. I am very skeptical about claims that computers can replace teachers. However, if computers can – and can do a better job – they should. We should always be looking for ways to improve not only the quality of post-secondary education, but also its cost-effectiveness. One can argue about the level of investment needed, but given the challenges on a global basis, we should not ignore opportunities to stretch scarce resources – and in particular skilled teachers – further.

These articles in fact are very interesting in that between them they lay out different roles for (human) teachers and computers. In most of the papers, the role of computers or software is to enhance or make more effective the role of teachers, rather than replacing them; in other words, the information science approaches here are providing additional tools for instructors.

There are several reasons for this. Perhaps the most important is that many of the tools or approaches described here are still in the early stages of development. They are partly developing definitions, theories and new approaches, and partly testing them as prototypes. Teachers are still needed, to provide input, to validate and to test the prototypes. We don’t know if some of the approaches set out in these papers will eventually be feasible or will work when scaled up. Even if the tools do turn out to be effective and scalable, the authors often see these tools as requiring additional intervention or control by teachers, and this is likely to hold for a long time.

Another reason is the still very strong limitations of computing in dealing with semantics, meaning, context and complexity. Despite huge advances in computing power, developing ontologies or protocols that apply to the extraordinarily wide range of contexts and variables in which most learning occurs is extremely challenging. One way this is done is to break the challenge into sub-sets, with the rest left to the training, experience and intuition of ‘live’ teachers. Reading these papers, it seems that the sub-sets being dealt with, while helpful, are still somewhat on the fringes of the challenges faced in most learning contexts. However, they are a start, and several (for instance recommendation systems for identifying papers most helpful for a particular learning task) seem extremely promising.

The third reason why this remains such a challenge (although one that is the easiest to deal with) is the very narrow view of learning often held by the computer scientists who work in this field, who tend to focus (not surprisingly) on teaching as information transmission and retrieval, rather than on teaching as cognitive, personal and social development. One reason of course is that it is easier to develop ontologies for the former and extremely difficult for the latter. Too narrow a view of learning is an easier challenge to overcome because it should not be difficult to ensure that computer scientists and educators work together as equals in approaching the challenge of teaching and learning. While most of the papers in this edition did seem to embrace this broader approach to learning, some did not.

Nevertheless, I greatly appreciate IRRODL’s decision to focus on this area, because we do need to bridge the world of computer scientists and educators if the power of computing is to be wisely applied to education and training.

Implications of the papers – especially for MOOCs

Once some of these approaches are established and validated, their main value is that they can be scaled up. This is of particular significance to MOOCs. Currently the main challenge for MOOCs is to:

  • find ways of automating learner interaction with materials beyond the level of checking that information has been retained
  • provide contextually rich feedback on learning
  • improve unsupervised peer interaction to ensure knowledge construction,
  • avoid, detect and deal with plagiarism
  • provide secure forms of authentic and valid assessment of learning,

all on a massive scale.

In these papers, we did see some of the ways in which these problems might be resolved, or at least a more general approach to dealing with large blocks of learners with few instructors. I suspect that over the next year or so, we will see similar developments being applied to the design of MOOCs. How effective such approaches will be remains to be seen, but there is promise and it certainly seems worth trying. I just hope though that those responsible for MOOCs will apply as rigorous evaluation protocols as are found in the papers in this edition. Let’s hope that this is at least combined with independence in the evaluation of MOOCs.

Which way for online learning?

Lastly, I’m wondering whether we will see two very divergent approaches to online learning, one based on very low cost or free teaching to massive numbers, drawing heavily on a computer science approach to teaching and learning, and another based on a more humanistic approach to teaching and learning, with smaller numbers and greater involvement of human teachers, and hence much more expensive, but still with greater focus than at present on hybrid and fully online delivery.

In my mind, I think (or rather hope) that it will be neither such extremes, but a mix of the two approaches. Good quality education is never going to be free for most people; there will always be costs. The human approach will also remain a core component for most education. But a judicious combination of computer science and humanistic designs and flexible delivery should enable high quality education to be delivered much more cost-effectively than it is at present. These articles are important milestones on this journey.

Summary of papers in this edition

Butakov, S. et al. (2012) Protecting Students’ Intellectual Property in the Web Plagiarism Detection Process IRRODL, Vol. 13, No. 5

This article suggests an architecture for plagiarism detection that protects the student IP by sending a randomized selection of content to a third party plagiarism detector.

Yu, P-T. et al. (2012) A Rapid Auto-Indexing Technology for Designing Readable E-Learning Content IRRODL, Vol. 13, No. 5

This paper presents an automatic method for detecting the changes in a PowerPoint based videoed lecture, and embedding this technology in an online course as an interactive component.

‘The fastest and easiest way to provide an adequate amount of e-learning content is to record teachers’ presentations in a classroom or studio and then directly put those recordings into a learning management system (LMS)’…..However, this kind of streaming data lacks flexibility and interactive capability. Therefore, a user-friendly interface is required to let students easily capture any segment of the recorded instructional videos’

The authors designed a mechanism of regular testing which requires learners to answer questions corresponding to pop-up information triggered when they click on an access point found by the indexing mechanism. Changes in the powerpoint slides in this case acted as the trigger for the access point.

Cheng, J.-S., Huang, E. and Lin, C-L.  (2012) An E-Book Hub Service Based on a Cloud Platform IRRODL, Vol. 13, No. 5

This research project developed an e-book hub service on a cloud computing platform in order to overcome the limitations of computing capability and storage capacity that are inherent in many mobile devices. The e-book hub service also allows users to automatically adjust the rendering of multimedia pages at different resolutions on terminal units such as smartphones, tablets, PCs, and so forth.

Winoto, P., Ya, T, and McCalla, G. (2012) Contexts in a Paper Recommendation System with Collaborative Filtering IRRODL, Vol. 13, No. 5

The authors designed, developed and evaluated a recommender system (RS) that enables students to recommend papers that will facilitate other students in their learning. The RS was tried out on both ‘novice’ (undergraduate) and ‘experienced’ (post-graduate) students. The authors found that a multi-dimensional system that took account of different pedagogical factors worked better than a unidimensional RS based on ‘liking’.

Baldiris, S. et al. (2012) Searching for and Positioning of Contextualized Learning Objects, IRRODL, Vol. 13, No. 5

This paper focuses on two ways to increase the re-usability of learning objects (LO). The paper

‘promotes LO reuse by encouraging instructors to access distributed learning object repositories (DLOR) as sources of LO with diverse granularity that could be elements in a generated learning design. [The] proposal consists of two different parts: the distributed learning object metadata searching process (LORSE) and the micro-context-based positioning process (LOOK).‘

The authors found that to achieve a viable solution with these repositories, the object metadata (in the LO depositories investigated) needs to be refined. Metadata available in the involved repositories currently has limited information. This inhibits identifying the contextual relevance of a learning object for re-use in a learning design.

Wen, D., Cuzzola, J., Brown, L. and Kinshuk (2012) Instructor-Aided Asynchronous Question Answering System for Online Education and Distance Learning IRRODL, Vol. 13, No. 5

This paper introduces a question answering (QA) system particularly suited for delayed-answered questions that are typical in certain asynchronous online and distance learning settings. The authors propose a solution that integrates into an organization’s existing learning management system. They present how their system fits into an online and distance learning situation and how it can better assist supporting students.

Wong, W-K., Yin, S-K, and Yang, C-Z (2012) Drawing Dynamic Geometry Figures Online with Natural Language for Junior High School Geometry IRRODL, Vol. 13, No. 5

This paper presents a tool for drawing dynamic geometric figures by understanding the texts of geometry problems. With the tool, teachers and students can construct dynamic geometric figures on a web page by inputting a geometry problem in natural language. A preliminary evaluation of the tool showed that it produced correct dynamic geometric figures for over 90% of problems from textbooks. With such high accuracy, the system produced by this study can support distance learning for geometry students as well as distance learning in producing geometry content for instructors.

Nguyen, B-A., and  Yang, D-L. (2012) A Semi-Automatic Approach to Construct Vietnamese Ontology from Online Text IRRODL, Vol. 13, No. 5

An ontology is an effective formal representation of knowledge used commonly in artificial intelligence, semantic web, software engineering, and information retrieval. The authors present a support system for Vietnamese ontology construction using pattern-based mechanisms to discover Vietnamese concepts and conceptual relations from Vietnamese text documents. The approach provides a feasible solution to build Vietnamese ontologies used for supporting systems in education.

Tierney, P. (2012) A Qualitative Analysis Framework Using Natural Language Processing and Graph Theory IRRODL, Vol. 13, No. 5

This paper introduces a method of extending natural language-based processing of qualitative data analysis with the use of a very quantitative tool—graph theory. It is not an attempt to convert qualitative research to a positivist approach with a mathematical black box, nor is it a “graphical solution”. Rather, it is a method to help qualitative researchers, especially those with limited experience, to discover and tease out what lies within the data.

Filed Under: Bibliography, cheating, course design, developing intellectual skills, e-books, evaluating courses and programs, Events, knowledge-construction, learner support, learning objects, lectures, mobile learning, MOOCs, open content, open source, plagiarism, publications, research, search engines, Teaching and learning, Technology, video Tagged With: , , , ,

Online learning in 2012: a retrospective

December 16, 2012 By 17 Comments
Listen with webReader

© The Greening of Gavin, 2012

Well, 2012 was certainly the year of the MOOC. Audrey Watters provides a comprehensive overview of what happened with MOOCs in 2012, so I won’t repeat what she has done. Instead in this post I will focus mainly on trying to explain with regards to MOOCs what appears to me to be highly irrational organizational behaviour, more akin to lemmings than pillars of higher learning.

Why MOOCs?

For those of us who work mainly in universities and colleges, the hype around MOOCs is like living in two parallel universes: what we do every day in online learning, and what we read or hear about in the media. (I leave you to judge which is the true reality.) Even organizations that should know better think that online learning started at MIT in 2002 with OpenCourseWare. So why have MOOCs in particular got so much press?

This is an exercise in social anthropology.

To quote from Wikipedia:

It is unknown why lemming populations fluctuate with such variance roughly every four years, before plummeting to near extinction.

Now some evidence suggests their predators’ populations, particularly the stoat, may be more closely involved in changing the lemming population

Lemmings can swim and may choose to cross a body of water in search of a new habitat. In such cases, many may drown if the body of water is so wide as to stretch their physical capability to the limit.

 I believe there are several themes that have led to MOOC hysteria in 2012:

  • they appear to be free. The direct costs of higher education, especially but not only in the USA and the UK, have been systematically transferred from the tax payer to the individual student or parents through cuts in government funding and increases in tuition fees. In other words, the cost of higher education has become more transparent. It’s really expensive. Free of course is better than expensive. MOOCs have been promoted as being free. However, there are no free services. All services have a true cost. At least to date, MOOCs are the opposite of transparency on the true cost. We do know that over a hundred million dollars have been invested this year alone in MOOCs, but what are the costs of the professors’ time, the cost of managing large numbers of students, and above all, the cost of ensuring student learning (however it is measured)? We just don’t know. Until we do, it’s a shell game
  • it’s also a numbers game: input matters more than output. The focus of the media has been on the massive numbers enrolling. However, there has been little focus on what students are actually learning. All we know is that completion rates are pathetic (less than 10%), and many of those that do complete are already well educated. Nevertheless it is argued that on a global perspective, the completion numbers are still large. However, so are the numbers in traditional higher education, and also in credit-based online learning. Sloan and Babson have been tracking the online credit numbers for years. They have been growing at a steady rate of between 12-20% a year. Ontario alone has over 500,000 online course registrations in its public universities and colleges, with completion rates in the 75-85%, matching completion rates in face-to-face classes. Millions are taking online courses for credit in Asia. But does this get mass coverage in the media? No.
  • technology triumphs over teaching: MOOCs in general have been driven by computer scientists who believe that just ‘delivering’ content over the Internet equates to learning. It doesn’t, but broadcast content delivery is something that lazy reporters can easily understand.
  • it’s all about the elite institutions. The media love to focus on the ivy league universities to the almost total neglect of the rest of the system (the cult of the superstar). Here is an appalling irony. The top tier research universities have by and large ignored online learning for the last 15 years. Suddenly though when MIT, Stanford and Harvard jump in, all the rest follow like lemmings. MOOCs are seen as an easy, low risk way for these universities not only to catch up, but to jump into the front line. But they are hugely wrong. Moving from broadcasting to learning is not going to be easy. More importantly, MOOCs are a side issue, a distraction. The real change for universities is going to come from hybrid learning – a mix of on-campus and online learning. Those top tier research universities though are going to miss out on this, by sidelining their online learning to a peripheral, continuing education activity.
  • don’t forget the politics: There’s just been a presidential election in the USA. A number of corporate leaders and some in the Republican party want to privatize the US higher education system. Anything that will undermine it is heavily promoted. MOOCs to some extent have been a tool in the hands of the media for suggesting that education need not be expensive and could be ‘free’, or at least much lower cost, if left to business. This fits the agenda of the right.

Having said all this, I believe that there is a future for MOOCs, but that’s for another post, my outlook for 2013, which comes in January.

In the meantime, there were, believe it or not, several other interesting developments in online learning, but before exploring those as well, let’s see how right I was in my outlook for 2012.

What I predicted

  1. The year of the tablet: 99% probability
  2. Learning analytics: 90% probability
  3. Growth of open education: 70% probability (depending on definition of open education)
  4. Disruption of the LMS market: 60% probability
  5. Integration of social media into formal learning: 66% probability
  6. The digital university: 10% probability
  7. Watch India
  8. The great unknown: 10% probability

Well, not a great record at prediction. I suppose you could include MOOCs within ‘growth of open education’. But look at what I actually wrote:

open access to high quality (all right, highly qualified) instructors is likely to be limited to idealistic volunteers, or to limited events (e.g. a MOOC), mainly because of a mis-match between supply and demand. Too many people want access to what they may incorrectly assume to be high quality instructors at elite institutions, for instance. This is partly an institutional barrier, as institutions try to protect their ‘star’ faculty, which is why this form of openness depends largely on individual volunteers.

Not actually wrong, but it certainly didn’t capture the mania that would develop around MOOCs in 2012.

Although there have been lots of interesting individual uses of tablets, particularly in k-12, they certainly haven’t taken off to the extent to which I predicted, at least in post-secondary education. However, so much in prediction depends on timing – maybe it will happen this year. For instance, mobile learning, one of my predictions for 2011, certainly expanded in many institutions in 2012, and will certainly continue to grow in 2013. The use of data analytics definitely increased, but still in a minority of institutions, in 2012, but learning analytics are still being used by a very tiny minority. The technology isn’t quite ready yet. (Again, this depends on definition – I’m talking about the hope that learning analytics will help instructors to achieve better learning outcomes, or put another way, will help students to improve their learning.)

What you read

Another way at looking at 2012 is to see what you chose to read. There are just over 1,800 posts on the site. Here are the top 14 posts in 2012, with the number of hits. (If you missed one, just click on it.)

Recommended graduate programs in e-learning

15,685
What’s right and what’s wrong about Coursera-style MOOCs

7,089
e-learning outlook for 2012: will it be a rough ride?

6,827
New technologies for e-learning in 2012 (and a little beyond)

6,658
A short critique of the Khan Academy

5,026
Can you teach ‘real’ engineering at a distance?

4,988
What Is Distance Education?

4,083
Why learning management systems are not going away

3,624
E-learning quality assurance standards, organizations and research

3,221
A personal view of e-learning in Saudi Arabia

2,844
A student guide to studying online

2,513
10 types of plagiarism (and why I’m pleading guilty to at least one charge)

2,353
Daniel’s comprehensive review of MOOC developments

2,264
Designing online learning for the 21st century

1,929

The numbers of course are skewed by their date of  posting. Those posted early in the year have more chance of being accessed than those posted later. Timing also matters in terms of external events. Despite all the hype about MOOCs, only two of the top 14 posts were specifically on MOOCs (although there were several others posted). I am though surprised at the amount of interest in prediction, especially given how bad I am at it!

The inclusion of ‘Can you teach real engineering at a distance?’ at no. 6 is really interesting. This was posted originally on July 5, 2009, but it has sustained a long discussion that is still active today. I was also pleased to see that designing online learning for the 21st century squeezed in, as this was about design of online learning. I’m glad there’s still at least some interest in this issue. There is also evidence that the site is being used by  a lot of online students (or potential students), which is very gratifying. I need to do more posts targeted to students next year.

What I did

Since I’m not free and open (except here), this is some indication of what institutions were interested in this year (at least enough to pay me for it).

Site visits for consultancies or discussions with faculty/staff on strategies or designs for online learning

  • Mexico City: to develop a business plan for a national Mexican virtual university
  • Edmonton: Campus St-Jean, University of Alberta: informal review of online learning activities
  • Université de Sherbrooke, l’université Laval and Université de Québec en Abitibi-Témiscamingue, Québec
  • Vancouver Community College, Kwantlen Polytechnic University, and University of British Columbia, BC
  • University of Manitoba, Winnipeg
  • EFQUEL conference, Granada, Spain
  • COHERE conference, Calgary, Alberta

Online consultancies

MOOCs and Webinars

  • planning and managing online learning: participant in #Change 11 cMOOC
  • costs of online learning: guest instructor for University of Maryland University College/University of Oldenberg, Germany
  • Elections Canada: online course design

Institutional site visits and reports on gamechanging institutions

  • Western Governors University
  • Open University, UK
  • Open University of Catalonia, Spain
  • London Knowledge lab, Institute of Education, London, UK.

It can be seen there was a great deal of interest in:

  • strategies and management,
  • new course designs,
  • design and organization of online institutions,
  • the costs of online learning

during 2012. These issues are not likely to disappear next year, either.

Politics and economics

In 2012, there were major developments in both the politics and economics of online learning. Governments in the USA and Europe accelerated cost cutting in post-secondary education. Nearly one billion dollars has been cut from the community college system in California alone since 2008. Student tuition fees have risen dramatically over the last five years in both the USA and the U.K. Even in Canada, provincial governments are facing the need to constrain public funding.

In Ontario, Canada’s largest province, the government threw down a challenge to the post-secondary institutions. Enrollments will need to increase, quality must be obtained, but there will be no new money. What can the institutions do to increase productivity through innovation? It’s a good question. Business cannot go on as usual. There is surely room for improvement and change in our institutions.

This theme is likely to continue into 2013. Governments, parents and increasingly students will be looking to online learning to increase productivity: better learning outcomes for less money. Are we up to the challenge?

Goodbye, 2012

I asked the question last year: will it be a rough ride? It’s certainly been a fast ride and quite bumpy at the same time. I don’t know how you feel, but I feel I’m hanging on, but only just. It’s good though that it’s exciting, stimulating, infuriating, and frustrating. It means that online learning is alive and well, growing in both breadth and more importantly depth.

So to all my readers, thank you for coming along for the ride. Have a great break, merry Christmas, happy Hanukah, or just have a good time, whatever your religion or beliefs. And I look forward to sharing my outlook for 2013 in the new year.

Questions

1. What pleased, surprised or disappointed you in 2012 with regard to online learning?

2. What do you think was the most important development in 2012 for online learning? Obama’s re-election? MOOCs? New course designs? Or something else?

3. Are we up to the challenge of using online learning to increase productivity through innovation? If so, what would that look like?

Filed Under: analytics, Asia, blended learning, Blogs, Canada, China, completion rates, costs and benefits, course design, current state of the art, design of virtual learning organizations, e-learning, e-learning and distant education, economic impact, engineering, England and Wales, Europe, evaluating courses and programs, futures, General, Google wave, India, informal/non-formal learning, innovation, iPads, learner support, learning management systems, mobile learning, Monthly Feature, MOOCs, Ontario, open content, planning, policies and management - institutional, post-secondary distance education, Strategies, planning and management, tablets, Teaching and learning, Technology, Tony's Blog, Tweets, United Kingdom, United States of America, USA Tagged With: , , , , , , , , ,

What’s right and what’s wrong about Coursera-style MOOCs

August 5, 2012 By 52 Comments
Listen with webReader

 

TED Talks: Daphne Koller: What we’re learning from online education

Daphne Koller, one of the two founders of Coursera, describes some of the key features of the Coursera MOOCs, and the lessons she has learned to date about teaching and learning from these courses. The video is well worth watching, just for this.

However I’m probably going to suffer the same kind of fate of the Russian female punk band, Pussy Riot, by spitting on the altar of MOOCs, but this TED talk captures for me all that is both right and wrong about the MOOCs being promoted by the elite US universities.

Let me start by saying that I actually applaud Daphne Koller and her colleagues for developing massive open online MOOCs. Any attempt to make the knowledge of some of the world’s leading experts available to anyone free of charge is an excellent endeavour. If only it stopped there.

What I object to is the hubris and misleading claims that are evident in this TED video. As someone once said about one of Sigmund Freud’s lectures, what is new is not true, and what is true is not new.

Myth 1: MOOCs increase access to higher education in developing countries

She starts by using the example of students being trampled to death trying to get into the line for the very few places left open by the campus-based University of Johannesburg in South Africa. This is a particularly maladroit example. Yes, there is a desperate shortage of conventional university places in South Africa. But South Africa has probably the oldest distance and open teaching university in the world, UNISA, currently with over 160,000 students. Just providing not for credit open online learning from the USA will not solve South Africa’s access problems (especially as most of those seeking university places do not have home Internet access). Indeed, to suggest that Coursera is an alternative to conventional university education takes the pressure off governments such as South Africa’s to find their own, indigenous solutions to access to higher education.

If Stanford or MIT gave credit for these courses to students from South Africa who succeeded in the exams, and then awarded them full degrees, then that might be different. But these elite universities continue to treat MOOCs as a philanthropic form of continuing education, and until these institutions are willing to award credit and degrees for this type of program, we have to believe that they think that this is a second class form of education suitable only for the unwashed masses.

Myth 2: new pedagogy

Second, the teaching methods used by most of the Coursera courses so far are based on a very old and outdated behaviourist pedagogy, relying primarily on information transmission, computer marked assignments and peer assessment.  Behaviourist pedagogy has its value, especially where there are right and wrong answers, facts or procedures that must be learned, or students lack higher level cognitive processing skills. In other words it works reasonably well for certain levels of training. But it is extremely difficult if not impossible to teach higher order skills of critical thinking, creative thinking, and original thinking using behaviourist pedagogy, the very skills that are needed in a knowledge-based society. (It should be noted that the ‘Canadian’ MOOCs of Stephen Downes, George Siemens and Dave Cormier do not suffer from this fault).

Third, and this is the most enraging part of the presentation for me, Daphne Koller talks as if she invented online learning, and that nothing was known beforehand about works and doesn’t work in online learning. So she has discovered that students learn better if they are active, so there are lots of tests and activities in the courses. It is better to break up monolithic one hour lectures into smaller, more digestible chunks. Both these strategies in fact date back to the UK Open University print packages forty years ago and it has been standard practice to incorporate such strategies in most online learning since it began on a serious scale 20 years ago.

Her comparisons are all with the weaknesses of lecture-based teaching. For this we should perhaps be thankful but again this is not new – online educators have been making this point again for over 20 years. And now Coursera is creating local or online study groups: again standard practice in other forms of online learning.

Myth 3: big data will improve teaching

One example used in the video was how computer-tracking of student activities can identify weaknesses in the teaching. The example was over 2,000 students giving the same wrong answer to a multiple choice question. In other words, Coursera is using trial and error as a form of teaching: try something, and if it doesn’t work, correct it the next time round. However, if they followed good design principles from the outset – for instance working with an instructional designer who could spot such errors or pre-testing material before it goes out to hundreds of thousands of guinea pig students – many of these ‘errors’ in teaching would be avoided in the first place. It is far, far better to avoid errors in teaching than to try to correct them afterwards: unlearning is much harder. With massive numbers of online students, the negative impact is equally massive.

Myth 4: Computers personalize learning

No, they don’t. They allow students alternative routes through material and they allow automated feedback but they do not provide a sense of being treated as an individual. This can be done in online learning, but it needs online intervention and presence in the form of discussion, encouragement, and an understanding of an individual student’s needs. The TED lecture omitted any discussion of completion rates. Again, this should not be the measure of MOOCs, but if you are going to argue that this form of teaching is superior to other forms of online learning, then discussion of completion rates becomes valid.

Daphne Koller’s final comment is telling:

‘We should spend less time at universities filling our students’ minds with content by lecturing at them, and more time igniting their creativity … by actually talking with them.’ 

However, that requires the presence of a teacher, either in the class or online.

Conclusion

I am sad having to write this. Daphne Koller gave a good lecture. Even these MOOCs are valuable, because, coming from elite universities, they have woken up the media in particular, and brought online learning to the attention of the public. I believe MOOCs have great potential for higher education: but not these MOOCs. And please, is it too much to ask for a little humility? (Probably, from so-called elite institutions).

Lastly, be careful what you wish for. Underlying all this is a fundamental question: is online learning best left to computer scientists or to teachers (or even students)? I know where I stand on this. What about you?

Some other reading

See also Lloyd Armstrong’s excellent post: Coursera and MITx: sustaining or disruptive? Changing Higher Education, August 6

What should we do about MOOCs? The Board Discusses

Is online learning really cracking open the public post-secondary system? 

Thoughts on the pedagogy of Coursera-style MOOCs 

Qué está bien y qué está mal en el estilo MOOC- Coursera (Spanish translation):

 

 

Filed Under: accreditation, analytics, assessment, Blogs, course design, creativity, critical thinking, design of virtual learning organizations, developing intellectual skills, e-assessment and e-portfolios, evaluating courses and programs, informal/non-formal learning, international and cultural issues, knowledge-construction, learner support, learning theories, Monthly Feature, MOOCs, quality and quality assurance, Strategies, planning and management, Teaching and learning, Tony's Blog, Tweets Tagged With: , , , , ,

“What should we do about MOOCs?” – the Board of Governors discusses

July 22, 2012 By 8 Comments
Listen with webReader

© Park Ridge Underground 2012

By accident, a microphone was left switched on at a medium sized state university’s Board of Governors meeting last week. In the best tradition of WikiLeaks, I am pleased to share a transcript with you.

Madam Chair. ‘Next item. Mr. Gedrich asked for this item to be included on the agenda. As a result, I have asked President Grimface to provide a briefing to the Board on MOOCHes’

President Grimface: ‘MOOCs, madam chair.’

Madam Chair: Ah, yes. Over to you, President Grimface.’

President Grimface. I think everyone here today is well aware that 10 top universities in the USA, and two from foreign countries, have joined an organization called Coursera, that is planning to offer massive open online courses, or MOOCs, to hundreds of thousands of students around the world. In addition, MIT and Harvard have created an organization called edX, and two Stanford professors have created their own organization called Udemy, all more or less to do the same thing.

There are no entry requirements, anyone can take the courses, in most cases they are free to the learners, and participants can even take an exam at the end for a certificate or letter of completion from the university. However, these courses do not count for credit, and the value of the experience is entirely dependent on the quality of the content and the name of the university by association. Nevertheless, there are indications that many employers, desperate for workers with knowledge in specific areas, will look kindly on learners with such qualifications.

Mr. Gedrich: Well, why aren’t we offering MOOCs then?

President Grimface: A good question, Mr. Gedrich. As a result of your request for a briefing, I have consulted widely within the university. There are in fact a variety of views about whether we should get involved, so before I come to a recommendation, I would like to lay out some of the pros and cons.

We have, as you know, a number of idealistic professors who believe that MOOCs will enable millions of people free access to the quality content that we offer. The way MOOCs are designed means that one professor can handle an unlimited number of learners, with the help at most of a couple of teaching assistants.

Mrs. Dowtful: But how does the poor man mark all the assignments?

President Grimface: Ah, but that’s just it, Mrs. Dowtful. The computer does the marking. The computer also collects data on all the student responses to the computer-generated questions, so the professor can see what most students have understood and what they haven’t, so it can be corrected or improved the next time round. For other questions that do not have such clear, correct answers, other learners provide feedback to those that get things wrong or don’t understand, through an online Facebook-type page. With so many learners on a course, somebody is bound to know the answer.

Mrs. Dowtful: But if they already know the answer, why are they taking the course?

President Grimface: Well, although some students will already be familiar with some parts of the course, there will be other parts that are new to them. And some students may already have learned the same material elsewhere, but they haven’t got a letter of completion from Stanford or Harvard.

Dr. Hardnose. This all sounds terribly risky to me. How many people end up with certificates or letters of completion?

President Grimface: Well, for the first courses, less than 5% of those who started actually completed, but these courses are very new.

Dr. Hardnose: Don’t we offer online courses already? And what are the completion rates for those?

President Grimface: I’m glad you asked that, Dr. Hardnose. We do in fact have over 300 online courses, both at undergraduate and graduate level, and the completion rates are around 85%, almost the same as for our classroom-based courses. But these courses are part of degrees, for credit, and students have to meet our admission requirements. We also limit the class size to a maximum of 30 per instructor, so that there can be lots of online interaction, discussion and feedback between the instructor and the students. We’ve found that’s essential otherwise students don’t complete.

Madam Chair: It sounds to me that we are talking about completely different things here. The MOOCs – is that right? – are really non-credit courses and since the universities don’t take money off the students, and don’t give formal credit or degrees, it really doesn’t matter if many students don’t finish the courses.

Dr. Hardnose. What’s the point then? They might as well just wander into a library. And don’t the students feel cheated if they do lots of work and still don’t understand? Just because it’s offered by some Stanford professor doesn’t necessarily make it a satisfying educational experience.

President Grimface: I think that’s a bit hard. These courses are very new. Mistakes are inevitable to start with. The main thing is that  the content is often simply not available from anywhere else. These are often the main experts in the field, drawing on their latest research. That’s part of the attraction for many of those who sign up.

Mr. Gedrich: Exactly. Don’t we have some leading researchers and experts? Why aren’t they sharing it with everyone?

Dr. Hardnose. But if the university did that, these professors wouldn’t have time to teach their own students. In fact, didn’t some of the professors quit Stanford to set up their own business around MOOCs?

Mr. Gedrich: That’s my point. Why isn’t the university using MOOCs to generate more money?

Mrs. Dowtful: But aren’t they free?

Mr. Gedrich: Ah, Mrs. Dowtful, that’s where having a business brain comes in. You don’t sell the steak, you sell the sizzle.

Mrs. Dowtful: I beg your pardon?

Mr. Gedrich: Look, you’ve got hundreds of thousands of people ‘hooked’. You give them the course free, but you sell advertizing (think Google), or you hook them up with employers desperate for quality employees, who pay the university a commission for getting access to these people (particularly the several thousand who still manage to get a letter of completion). Or just charge the students a stiff fee for the exam at the end. After doing all that hard work, they’ll cough up. There’s lots of ways you can make money on this stuff – and it’s free education! Think what it will do for the university brand – you’ll look good and make money at the same time.

Madam Chair: I don’t want to stop discussion but I’m conscious that we have a full agenda today, and we’re already running late. I believe you have a proposal, President Grimface.

President Grimface: Several of my leading professors have come forward offering to run MOOCs. They like the idea of reaching out to thousands of learners. Some of them already have an online course as part of a degree program, and it wouldn’t be difficult to adapt them so that they could run as a MOOC as well. Some of the other professors don’t yet have an online course, but are willing to move a lot of their classroom material and videos of lectures online as a MOOC, then get the credit students to follow the MOOC and come to class to discuss the topics in the MOOC.

The real problem is assessment or testing at the end of the course.  Some of the professors are from Arts, Social Sciences and Business, and don’t feel computer marking is the best way to test students’ knowledge. We did a little research and found that there’s another kind of MOOC developed by some Canadians that is based more on discussion topics set by the instructor around the lectures and readings but which the MOOC learners contribute to and manage themselves. Indeed they can even put their work up and get other students to give a grade for it, so they would get some idea of how well they are doing.

So I’m thinking that we should set up a web site that lists all our intended MOOCs, have a simple log in process where participants register online for a particular course, and then once all the content, and activities like discussion topics, are loaded, we just assign a post doc graduate teaching assistant to monitor the MOOC, with the professor stepping in if there’s a problem. We could start  almost immediately, by the start of the next semester at the latest, with 12 MOOCs. We would give participants who ask a letter of completion, but it would carry no value for credit or admission here.

Another reason I’m in favour of this is the knock-on effect it may have on our credit programs. Although we have 300 online credit courses already, this is less than 10% of the total. We’ve found it very hard to get the majority of faculty to move into online learning, even though student demand appears to be strong. Already some faculty who have resisted online credit courses have shown interest in running a MOOC, so this may provide a back door into more online credit programs – a sort of reverse move, from non-credit to credit.

Dr. Hardnose: How much is this going to cost, and where’s the money going to come from? Didn’t I hear that MIT and Harvard are throwing $60 million at developing MOOCs?We’re all aware of the difficult financial situation the university is in at the moment.

Professor Grimface: This isn’t going to be a very expensive move, in the way we are thinking of it. The cost of the teaching assistants would be under $20,000 a year. The CIO and the Director of the Learning Technology Centre assure me that the technology challenges are more than manageable. We would need to spend some money on marketing but that would come out of the existing marketing budget. The professors are already paid and are wanting to do this, without disrupting their other teaching. It seems to us that this is a low risk, high gain strategy.

Mr. Gedrich But where’s the business plan? How are you going to make money in this wishy-washy way? You need to get someone in with a good business brain to set up and run this initiative.

Dr. Hardnose: For once I find myself agreeing with Gedrich, but for different reasons. I worry about how sustainable this is. It relies very heavily on the good will of a small number of professors. It’s bound to detract from their other teaching and even more, their research. You need to have a steady revenue stream to make this sustainable. And does this mean we will eventually move away from our successful, high quality, carefully designed model for our online credit courses to replace them with sloppily designed MOOCs?

And then there’s the law of diminishing returns. Sure the MIT and Stanford MOOCs are getting huge numbers, but there’s only a few of them at the moment and they are getting enormous publicity from mainstream media. With respect, although we’re a good, solid university, we don’t have that kind of marketing clout. Once everyone starts jumping on the bandwagon, and the novelty loses it’s edge, will we get the numbers for our MOOCs?

Madam Chair: (with a sigh): I agree there’s lots of pros and cons around this idea. However, Professor Grimface has come forward with what seems to me a very sensible proposal. We’ve had discussions in earlier meetings that the university needs to find ways to encourage more innovation in teaching and learning. This proposal will allow us to test the water without taking a large financial risk. Indeed, I don’t think we even need to put this to a vote. We’re not being asked to approve a budget proposal above the President’s discretionary limit, and I see the proposal fitting well within the spirit of the strategic plan.

Dr. Hardnose. Madam chair, if we’re going to rubber stamp this proposal, I think we should at least request a report back from the President within 12 months that provides us with some firm figures about costs – both direct and hidden – participant numbers and some evaluation of the quality of the learning from this exercise.

Mr. Gedrich. I support Hardnose on this, and would like to add that the President provides us with a proper business plan when he next reports back on this, otherwise we run the risk of missing out on a wonderful opportunity to bring in some money for this cash-strapped institution.

President Grimface: I will of course be more than willing in 12 months time to provide a full report to the Board on how our MOOC initiative has developed.

Madam Chair: Good. Thank you, President Grimface. Now the next item: the parking crisis on campus.

 

Filed Under: assessment, Blogs, change management, completion rates, costs and benefits, course design, evaluating courses and programs, governance, informal/non-formal learning, leadership, MOOCs, planning, policies and management - institutional, Strategies, planning and management, Teaching and learning, Tony's Blog, Tweets Tagged With: , , , ,

Nine steps to quality online learning: Step 9: Evaluate and innovate

July 9, 2012 By 5 Comments
Listen with webReader

© Hilary Page-Bucci, 2002

In this post I discuss the importance of evaluating each offering of an online course, how best to do this, and then the importance of maintaining and improving the course.

This is the last in a series of 10 posts on designing quality online courses. The nine steps are aimed mainly at instructors who are new to online learning, or have tried online learning without much help or success. The first nine posts (which should be read before this post) are:

Nine steps to quality online learning: Introduction

Nine steps to quality online learning: Step 1: Decide how you want to teach online

Nine steps to quality online-learning: Step 2: Decide on what kind of online course

Nine steps to quality online learning: Step 3: Work in a Team

Nine steps to quality online learning: Step 4: Build on existing resources

Nine steps to quality online learning: Step 5: Master the technology

Nine steps to quality online learning: Step 6: Set appropriate learning goals

Nine steps to quality online learning: Step 7: Design course structure and learning activities

Nine steps to quality online learning: Step 8: Communicate, communicate, communicate

A condensed version covering all the main posts in this series can be found on the Contact North web site: What you need to know about teaching online: nine key steps. (French version: Ce que le personnel enseignant doit savoir sur l’enseignment en ligne: neuf étapes clés‘)

The ten posts are also being translated into Portuguese by Professor Luis Roberto Brudna Holzle, Federal University, Brazil, available at Science Blogs: Nove passos para uma aprendizagem on-line de qualidade

Steps 1-8: Building a strong foundation

The emphasis in this series of posts is on getting the fundamentals of online teaching right. The discerning reader will have noted that there isn’t much in these posts about exciting new tools, MOOCs, the Khan Academy, MIT’s edX, and many other new developments in online learning. These tools and new programs offer great potential and we will discuss some of these in this post. However, it doesn’t matter what tools or revolutionary programs are being used, what we know of how people learn does not change a great deal over time, and we do know that learning is a process, and you ignore the factors that influence that process at your peril.

I’ve focused mainly on using LMSs, because that is what most institutions currently have, and they provide an adequate ‘framework’ within which the key processes of teaching and learning can be managed. But if you get these fundamentals right they will transfer well to the new tools and programs; if they don’t transfer well, such tools are likely to be a passing fad and will eventually die, because they don’t support the key processes that support learning. For example, MOOCs may reach hundreds of thousands of students, but if there is no suitable communication with or ‘online presence’ from an instructor, then most students will fail (as is the case at the moment). MOOCs will survive and grow if they can accommodate the core processes of clear learning outcomes, learner support, clear structure, management of student and faculty workload, etc.

The last key ‘fundamental’ of the teaching and learning process is evaluation and innovation: assessing what has been done, and then looking at ways to improve on it.

Why evaluation is important

This step isn’t specific to online teaching. It really applies to all forms of teaching. However, especially for instructors new to it, online teaching is different and therefore likely to be seen as higher risk. Online and distance learning is always held to a higher standard than conventional teaching, so more effort is required to justify their use. For tenure and promotion, it is important if you are teaching online to be able to provide evidence that the teaching has been at least as successful as your classroom courses. Online learning itself is continually developing. New tools and new approaches to teaching online are constantly coming available. They provide the opportunity to experiment a little to see if the results are better, and if we do that, we need to evaluate the impact of using a new tool or course design. It’s what professionals do. But the main reason is that teaching is like golf: we strive for perfection but can never achieve it. It’s always possible to improve, and one of the best ways of doing that is through a systematic analysis of past experience.

What to evaluate

In Step 1, I defined quality online learning very narrowly. It is outcomes based:

 By quality, I mean ‘Reaching the same level or better with an online course as for an equivalent face-to-face course.’ This has two quantitative critical performance indicators:

  • completion rates will be at least as good if not better for the online version
  • grades or measures of learning will be at least as good if not better for the online version.

On a qualitative level, I suggested one other criterion:

  • quality online learning will lead to new, different and more relevant learning outcomes that are better served by online learning.

So these are the minimum requirements. The first two are easily measured in quantitative terms. We should be aiming for completion rates for an online course of at least 85%, i.e. of 100 students starting the course, 85 complete by passing the end of course assessment (unfortunately, many classroom courses fail to achieve this rate, but if we value good teaching, we should be trying to bring as many students as possible to the set standard).

The second criterion is to compare the grades. We would expect at least as many As and Bs in our online version as in a classroom version. (I am assuming that students are taking the same exams, etc., whether they are in class or online, and are being marked to the same standards).

The third criterion is more difficult, because it suggests a change in the intended learning goals for a course that is delivered online. This might include assessing students’ communication skills, or their ability to find, evaluate, analyze and apply information appropriately within the subject domain, which are not assessed in the classroom version. This requires a qualitative judgement as to which learning goals are most important, and this may require endorsement or support from a departmental curriculum committee or even an external accreditation body.

However, even if we measure the course by these three criteria, we will not necessarily know what worked and what didn’t in the course. We need to look more closely at factors that may have influenced students’ ability to learn. We have laid out in the various steps some of these factors. Some of the questions to which you may want to get answers are as follows:

  • What learning outcomes did most students struggle with?
  • Were the learning outcomes or goals clear to students?
  • Was the teaching material clear and well structured?
  • Was the LMS easily accessible and available 24×7?
  • Did students behave in the online discussion forums in the way expected?
  • What topics generated good discussion and what didn’t?
  • Did students draw on the course materials in their discussion forums or assignments?
  • Did students make use of the podcasts?
  • How many students logged in to the webcasts and did these students do better or worse than those that didn’t?
  • Were the students overloaded with work?
  • Was it too much work for me as an instructor?
  • If so, what could I do to better manage my workload (or the students’) without losing quality?
  • How satisfied were the students with the course?

I will now suggest some ways that these questions can be answered without again causing a huge amount of work.

Analysis of a sample of exam answers will often provide information about course structure and the presentation of materials

How to evaluate factors contributing to or inhibiting learning on an online course

There is a range of resources you can draw on to do this, much more in fact than for evaluating classroom courses, because online learning leaves a traceable digital trail of evidence.

  • student grades
  • individual student participation rates in online activities, such as self-assessment questions, discussion forums, webinars
  • qualitative analysis of the discussion forums, for instance the quality and range of comments, indicating the level or depth of engagement or thinking
  • student assignments and exam answers
  • student questionnaires
  • online focus groups.

However, before starting it is useful to draw up a list of questions as in the previous section, and then look at which sources are most likely to provide answers to those questions.

One word about student questionnaires. Many institutions have a ‘standard’ student reporting system at the end of each course. These are often useless for the purposes of evaluating online courses. The questions asked need to be adapted to an online learning environment. However, because such questionnaires are used for cross course comparisons, the people who manage such evaluation forms are reluctant to have a different version for online teaching. Secondly, because these questionnaires are usually voluntarily completed by students after the course has ended, completion rates are often notoriously low (less than 20%). Low response rates are usually worthless or at best highly misleading. Students who have dropped out of the course won’t even get the questionnaire in most cases. Low response rates tend to be heavily biased towards successful students. It is the students who struggled or dropped out that you need to hear from.

I find small focus groups work better than student questionnaires, and for this I prefer synchronous tools such as Blackboard Collaborate. I will deliberately approach 7-8 specific students covering the full range of achievement, from drop-out to A, and conduct a one hour discussion around specific questions about the course. If one selected student does not want to participate, I try to find another in the same category.

In addition, at the end of a course, I tend to look at the student grades, and identify which students did well and which struggled. I then go back to the beginning of the course and track their online participation as far as possible (the next generation of learning analytics will make this much easier). I find that some factors are student specific (e.g. a gregarious student who communicates with everyone) and some are course factor specific, e.g. related to learning goals or the way I have explained or presented content. This qualitative approach will often suggest changes to the content or the way I interacted with students for the next version of the course. I may also determine next time to manage more carefully students who ‘hog’ the conversation.

Innovate

Usually I spend quite a bit of time at the end of the first presentation of an online course evaluating it and making changes in the next version, usually working with a trusted instructional designer. After that I concentrate mainly on ensuring completion rates and grades are at the standard I have aimed for.

What I am more likely to do in the third or subsequent offerings is to look at ways to improve the course that are the result of new external factors, such as new software (e.g. an e-portfolio package), or new processes (e.g. student-generated content, using mobile phones or cameras, collecting project-related data). This keeps the course ‘fresh’ and interesting. However, I usually limit myself to one substantive change, partly for workload reasons but also because this way it is easier to measure the impact of the change.

It is indeed an exciting time to be an instructor. In particular, the new generation of web 2.0 tools, including WordPress, new, instructor-focused ‘lightweight LMSs such as Instructure, open educational resources, mobile learning, tablets and iPads, electronic publishing, MOOCs, all offer a wide variety of opportunities for innovation and experiment. These can be easily integrated within the existing LMS and existing course structure. I will discuss in another post how some of these tools can radically change the design and delivery of online learning.

However, it is important to remember that the aim is to enable students to learn effectively. We do have enough knowledge and experience to be able to design ‘safe’, effective learning around standard LMSs. Many of the new web 2.0 tools have not been thoroughly evaluated in post-secondary educational settings, and  it is already clear that some of the newer tools or approaches are not proving to be as effective as older approaches to online learning. New is not always better. Thus for instructors starting in online learning, I would urge caution. Follow the experienced route, then gradually add and evaluate new tools and new approaches to learning as you become more experienced.

Summary

The nine steps are based on two foundations: effective learning strategies resulting from tested learning theories; and experience of successfully teaching online. The focus has been on instructors new to online learning. The posts are meant to lead you into working with other professionals, such as instructional and web designers, and preferably in a team with other online instructors.

The approach I have suggested is quite conservative, and some may wish to jump straight into what I would call second generation online learning, or e-learning 2.0. Nevertheless, even, or especially, working without a learning management system, it is important to remember that most students need clear learning goals, a clear structure or timetable of work, manageable study workloads, and instructor communication and presence. Most students also learn best, especially online, in a social environment that draws on and contributes to the knowledge and experience of other students.

Evaluation of the nine steps

In the spirit of this blog, it will help me evaluate the nine steps/ten posts of this series. So here are some questions:

  1. If you are new to online teaching, how helpful were these posts for you? What didn’t work, or what was missing?
  2. If you work with instructors who are new to or struggling with online teaching, would you refer these posts to them?
  3. Do you think this is a too conservative approach to teaching online? Too much focus on LMSs and not enough on web 2.0 tools?
  4. Do you agree that there are ‘fundamental processes of learning’ that are relatively independent of different tools?
  5. To what extent are these guidelines applicable to all kinds of teaching, not just online teaching? What do you think is ‘special’ that you need to know about online teaching?
  6. Was there something critical for quality online learning missing in the nine steps?

Your feedback either as a comment to this post or as an e-mail will be much appreciated and will make the next version much better!

Next steps

I will revisit in another set of posts two issues:

  • advanced online course design, based on the use of web 2.0 tools
  • the new campus: designing for hybrid learning

Further reading

I was surprised to find when conducting a mini review of formative evaluation in online teaching how little there is on the topic, and of what there is very little is helpful for the individual instructor trying to improve their course or could be recommended by me. If you know of any practical guide to formative evaluation of online teaching that will help individual instructors, please let me know! The best article by far that I found is:

Gunawardena, C., Lowe, C. & Carabajal, K. (2000). Evaluating Online Learning: models and methods. In D. Willis et al. (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2000 (pp. 1677-1684). Chesapeake, VA: AACE.

Note: there is a big difference between summative evaluation (which identifies the overall effectiveness for online learning compared with for instance classroom teaching) and formative evaluation, which seeks to learn from part experience to improve future performance. There is a large literature on summative evaluation of online learning, quality standards, and criteria. There is a much smaller literature specifically on formative evaluation of online teaching that enables an individual teacher to improve their teaching (although there is a much bigger literature on formative evaluation in classroom teaching). In this post, I have been focusing on formative evaluation, carried out by the instructor mainly to improve a specific course.

 

Filed Under: Blogs, course design, evaluating courses and programs, quality and quality assurance, Strategies, planning and management, Teaching and learning, Tony's Blog, Tweets Tagged With: , , , ,
« Older Posts
Newer Posts »