April 24, 2014

How problem-based learning can help develop innovation skills

Listen with webReader
© CTLT, UBC, 2013

© PBL Math, CTLT, UBC, 2013

Hoidn, S. and Kärkkäinen, K. (2014) Promoting Skills for Innovation in Higher Education A Literature Review on the Effectiveness of Problem-based Learning and of Teaching Behaviours Paris: OECD Education Working Papers, No. 100, OECD Publishing.

In a previous post, I discussed a report from the OECD that showed in a broad way the relationship between different pedagogies and the kinds of thinking associated with innovation. In particular it suggested that:

the emphasis of programmes on practical knowledge, on student-led projects and on problem-based learning are reflected in the level of creative skills, of oral communication skills and of teamwork and leadership skills of students.

Here I look at a second paper that explores in more detail the relationship between problem-based learning and skills found among innovators. In particular:

This report … reviews the current evidence on the effectiveness of problem-based learning compared with more traditional approaches in higher education teaching [and] explores the extent to which problem-based learning can be an effective way to develop different discipline-specific and transferable skills for innovation. 

Main results

  • Research, primarily from the field of medicine, shows that problem-based learning appears to be beneficial in fostering certain aspects of skills for innovation.
  • [In particular] …problem-based learning appears to be beneficial in fostering long-term retention and knowledge application, developing thinking and creativity skills, as well as social and behavioural skills (e.g. problem-solving, critical thinking, motivation, self confidence, team work).
  • By contrast, no clear difference between problem-based learning and traditional lecture-based teaching emerges as to performance in tests.
  • The benefits of PBL over traditional approaches seem to become more visible when examining higher education students’ long-term retention of knowledge. While PBL students may be slightly inferior to traditional students in overall knowledge and competence, they appear to be superior in long-term recall and retention.
  • Students in PBL appear to employ more productive approaches to study, have better interpersonal skills and appear to be more motivated than students in more traditional higher education programmes.
  • Despite the promising evidence linking problem-based learning and effective teaching in higher education to certain aspects of skills for innovation, more work is needed in this area. In reality there is no dichotomy between problem-based learning and “traditional” teaching and learning approaches  - policymakers and practitioners would benefit from a better understanding about which specific practices are effective for fostering different skill sets.
  • Faculty plays a pivotal role in enhancing student learning. Instructors can be trained to apply certain instructional behaviours that have been shown to be effective or to use student-centred forms of teaching and learning such as PBL and other methods that facilitate deep approaches to learning. Faculty can learn to give clear explanations and prompt feedback, present well-organised materials, ask students challenging questions, encourage student participation in the classroom and show concern and respect for students and student learning.

Implications for online learning

Although this paper does not discuss online learning or technology-based approaches to PBL, online learning can provide more flexibility and opportunities for problem-based learning, although to date, where problem-based learning and online learning have been combined, it is usually in a hybrid model. A common design is for students to gather in class for the definition of the problem, and instruction on the key steps to be taken, then to work collaboratively in small groups online on problem solution, returning to class for presentation and discussion of each group’s conclusions. However, there are also examples of fully online courses using a problem-based or inquiry-based learning approach.

In either hybrid or online learning modes, though, it is critical to give clear guidelines and structured steps to be taken to solving problems, especially for students who are new to this teaching approach. It is also important to ensure that assessment actually measures skills in problem-solving and critical thinking, and is not just a test of comprehension. Once again, it is not so much the mode of learning that matters as the quality of the teaching methods and assessment within that mode.

Comment

The study provides a pretty good overview of new developments in teaching in higher education, and where some of them are taking place. Indeed the paper recognizes that PBL started at MacMaster University in Canada in the 1960s. As the report notes:

[At that time] medical students lacked clinical reasoning, problem-solving, and critical thinking skills. There was concern that medical schools put a too heavy emphasis on memorisation of potentially irrelevant or soon-to-be-outdated facts instead of skills necessary to practice medicine. At the same time, medical students themselves seemed to be disenchanted and bored with their education because they had to absorb vast amounts of information of which much was perceived to have little relevance to medical practice.

The paper is worth reading, not so much for its conclusions, which are not startling, but because it provides an excellent summary on the research on how students learn at a higher education level, and the implications for the training of faculty. In essence, problem-based learning is valuable but depends on the learner having sufficient ‘foundational’ knowledge to enable them to tackle problems. This foundational learning may benefit from more traditional or formal approaches to teaching. The main value though of the paper is that it provides evidence-based guidelines for effective teaching.

 

A review of a Harvard/MIT research paper on edX MOOCs

Listen with webReader

 edX graphic

Ho, A. et al. (2014) HarvardX and MITx: The First Year of Open Online Courses Fall 2012-Summer 2013 (HarvardX and MITx Working Paper No. 1), January 21

This 32 page report provides a range of data and statistics about the first 17 MOOCs offered through edX by MIT and Harvard.

Methodology

MOOCs raise a number of interesting challenges when doing research, such as measuring participation, and defining success. In any interpretation of the results, these methodological challenges need to be considered. The researchers identified the following challenges:

1. Post-hoc research. The research design was established after the courses were designed and delivered, so data on some key critical research variables (e.g., socio-economic status) were not available or collected.

2. Variation in the object of the research. Although limited to MOOCs offered on the edX platform, the 17 MOOCs varied considerably in educational objectives, style, length, types of learner and other factors.

3. Measuring levels of participation. Participants varied from those who logged in only once to those that completed a certificate (and then some who went on to take more MOOCs). As a result, the researchers came up with four mutually exclusive categories of participation:

  • Only Registered: Registrants who never access the courseware.
  • Only Viewed: Non-certified registrants who access the courseware, accessing less than half of the available chapters.
  • Only Explored: Non-certified Registrants who access more than half of the available chapters in the courseware.
  • Certified: Registrants who earn a certificate in the course.

4. Percentages are misleading when numbers are large. This was a new one for me. I know one should never use percentages when n <20, specially when generalizing beyond the sample, but in this instance, the researchers argue that small percentages (e.g. <5%) are also misleading when the number the percentage refers to can be very large, e.g. when 3% = 1,400 students who completed a certificate. In such cases, the absolute numbers matter more than the percentage, so the researchers claim.

5. Measures of success The researchers argue that traditional measures of academic success, such as the percentage of those who successfully complete a course, are not valid (the word used is ‘counter-productive’) for open online courses.

Main results

Participation

  • 17 MOOCs
  • 841,687 course registrations: average per MOOC: 51,263
  • 597,692 ‘persons’: average of 1.4 MOOCs per person
  • 292,852 (35%) never engaged with the content (“Only registered”)
  • 469,702 (56%) viewed (i.e. clicked on a module) less than half of the content (“Only viewed”)
  • 35,937 (4%) explored more than half the content, but did not get a certificate (average per MOOC: 2,114)
  • 43,196  (5%) earned certificates (average per MOOC: 2,540)

Participants

  • 234,463 (33%) report a high school education or lower
  • 66% of all participants, and 74% of all who obtained a certificate, have a bachelor’s degree or above
  • 213,672 (29%) of all participants, and 33% of all who obtained a certificate, are female
  • 26 was the median age, with 45,844 (6%) over 50 years of age
  • 20,745 (3%) of all participants were from the UN listed least developed countries
  • there are ‘considerable differences in …. demographics such as gender, age… across courses.”

Comments

First, congratulations to Harvard and MIT for not only doing this research on MOOCs, but also for making it openly available and releasing it early.

Second, I agree that percentages can be misleading, a focus on certification is not the best way to assess the value of a MOOC, and that absolute figures matter for assessing the value of MOOCs. However, this is NOT the way most commentators and the media have focused on MOOCs. Percentages and certification DO matter if MOOCs are being seen as a substitute or a replacement for formal education. MOOCs need to be judged for what they are, a somewhat unique – and valuable – form of non-formal education.

Third, if we do look at absolute numbers, they are in my view not that impressive – an average of 2,540 per course earning a certificate, and less than 5,000 per course following more than half the content. The Open University, with completely open access, was getting higher numbers of students completing credit-based foundation courses when it started. The History Channel (a cable TV channel in North America) does a lot better, in terms of numbers. We have already seen overall average numbers for MOOCs dropping considerably as they have become more common. So when we account for the Hawthorne effect, the results are certainly not startling.

Fourth, these results so much reminded me of the research on educational broadcasting 30 years ago (for more details, see footnote). If you substituted ‘MOOC’ for ‘educational television’, the results would be almost identical (except there was a higher proportion of women than men participating). Perhaps they should read my very old book, “Broadcasting in Education: An Evaluation.” (I still have a few copies in a cupboard somewhere).

Lastly, this brings me to my final point. Where is the reference to relevant previous research or theory (see, for instance the footnote to this post)? There are certainly unique aspects to MOOCs that deserve to be researched. However, while MOOCs may be new, non-formal learning is not, nor is credit-based online learning, nor is open education, nor is educational broadcasting, of which MOOCs are a new format. Much of what we already know about these areas also applies to some aspects of MOOCs. Once again, though, Harvard and MIT seem to live in an environment that pays no attention to what happens outside their cocoon. If it’s not theirs, it doesn’t count. This is simply not good enough. In no other field would you get away with ignoring all previous research or work in related areas such as credit-based online learning, open education or educational broadcasting.

Having got that off my chest, I did find the paper well written and interesting and certainly worth a careful read. I look forward to reading – and reviewing – future papers.

Footnote: MOOCs and the onion theory of educational broadcasting

I eventually found a copy of my book. I blew the dust off it and guess what I found.

Here’s what I wrote about ‘levels of commitment’ in non-formal educational broadcasting in 1984 (p.99):

At the centre of the onion is a small core of fully committed students who work through the whole course, and, where available, take an end-of-course assessment or examination. Around the small core will be a rather larger layer of students who do not take any examination but do enrol with a local class or correspondence school. There may be an even larger layer of students who, as well as watching and listening, also buy the accompanying textbook, but who do not enrol in any courses. Then, by far the largest group, are those that just watch or listen to the programmes. Even within this last group, there will be considerable variations, from those who watch or listen fairly regularly, to those, again a much larger number, who watch or listen to just one programme.

Now compare this to Figure 2 (p.13) of the Harvard/MIT report:

MOOC onionI also wrote (p.100):

A sceptic may say that the only ones who can be said to have learned effectively are the tiny minority that worked right through the course and successfully took the final assessment…A counter argument would be that broadcasting can be considered successful if it merely attracts viewers or listeners who might otherwise have shown no interest in the topic; it is the numbers exposed to the material that matter…the key issue then is whether broadcasting does attract to education those who would not otherwise have been interested, or merely provides yet another opportunity for those who are already well educated…There is a good deal of evidence that it is still the better educated in Britain and Europe that make the most use of non-formal educational broadcasting.

Thanks for the validation of my 1984 theory, Harvard/MIT.

Reference

Bates, A. (1984) Broadcasting in Education: An Evaluation. London: Constable

 

Education across space and time: Distance Education, Vol. 34, No.2

Listen with webReader
Distance education in Australia

Distance education in Australia

This special edition of the Australian-based Distance Education journal presents a selection of papers originally submitted to the Open and Distance Learning Association of Australia’s 2013 summit meeting. The themes that the issue attempts to address are as follows:

  • How can we foster engaging and interactive learning with a dispersed and diverse population of students? 
  • How can we shift towards a learner-centred paradigm when institutional practices and physical infrastructures are geared towards teacher-centred delivery modes?
  • How can we enable the social and connected features of technology, when LMSs can be restrictive and clumsy…?

Sims, R. and Kigotho, M. (2013) Education across space and time: meeting the diverse needs of the distance learner:

This editorial sets the context and provides a brief description of each of the papers in the edition.

Hockridge, D. (2013) Challenges for educators using distance and online education to prepare students for relational professions

Relational professions are those which require ‘personal skills and a level of maturity.‘ This paper describes research that investigated educators’ concerns about distance and online education in Australian theological institutions. The paper in particular looks at ‘formation’, or character development, so the findings are more widely relevant than just theology. Her conclusion is well worth summarizing:

…it is overly simplistic to conclude that formational learning cannot occur in distance and online modes. Formational learning is complex and not easy to achieve regardless of the mode of study….a more productive way forward…is to be more intentional about the ways in which formation is addressed whether on campus, distance or online.

Earl, K. (2013) Student views on short-text assignment formats in fully online courses.

Short-text assignments restrict the word counts to 800 words or less. (Bit like a blog.) The study addressed two questions: how do students rate short-text assignments? How do students rate feedback provided by short text assignments? Conclusions:

assessment is more than a summative check of student knowledge and skills; it is an experience and part of the communication, and therefore relationship, between teachers and students. Short-text assignments are rated highly by students not because of a shorter word count but because students appreciated the variety and creativity aspects to these assignments. 

Note that the study was on one class of 21 students taught by the researcher.

Watson, S. (2013) Tentatively exploring the learning potentialities of postgraduate distance learners’ interactions with other people in their life contexts

Little consideration seems to have been given to the possibility that distance learners may be interacting with other people in their life contexts about their studies in a way that is making a positive contribution to their studies. The study involved semi-structured interviews of 15 Australian post-graduate students studying at a distance. Although the findings suggest that students vary widely in the extent to which they interact with others outside their course for study purposes, when they do interact, they produce identifiable learning benefits. Watson identified five types of life context interactions:

  • gathering information for assignments
  • getting help with difficult content
  • discussing the application of content to real-world contexts
  • sharing knowledge with others
  • getting feedback on assignment drafts

Watson suggests two course design implications from her studies so far:

  • encourage learners to talk to appropriate colleagues, friends or family about the application of particular theories in practice
  • encourage the establishment of mentoring relationships between learners and appropriate industry personnel

Higgins, K. and Harreveld, R. (2013) Professional development and the university casual academic: integration and support strategies for distance education

Casual academics are university instructors who are not entitled to either paid holiday leave or sick leave (such as, presumably, adjuncts and contract instructors in North America). In this study, twelve casual academics who taught distance education courses discussed their work through an in-depth semi-structured interview. The interviews revealed that these instructors managed their own professional development informally, and were sometimes unaware of the formal professional development activities available to them from the university.

Murphy, A. (2103) Open educational practices in higher education: institutional adoption and challenges

In this study, 110 individuals from higher education institutions in 29 countries participated in a survey aimed at identifying the extent to which HE institutions are currently implementing OERs and practices. The sample was focused on people with an interest in OERs; half the participants were from UK.

Main findings:

  • 23% were in organizations actively involved in the OERu network - 
  • 88% ‘knowledgeable’ about OERs
  • 29% were in institutions that were actively publishing OERs
  • the adoption of OERs and practices is still in its infancy
  • additional support such as funding and dedicated human resources are needed

Yasmin (2013) Application of the classification tree model in predicting learner dropout behaviour in open and distance learning

This study compares pre-enrollment student data with student attrition/drop-out for 12,000 post-graduate distance education students admitted to the University of North Bengal, India. The study indicated that married, employed, older, or remotely located students were more likely to drop out.

Note that the study used mainly demographic data, rather than data based on previous academic performance or the influence of factors during courses.

The paper’s main value is that it provides an analysis of drop-out factors for distance education students in a developing country, complementing the vast array of similar studies in developed countries.

Todhunter, B. (2013) LOL – limitations of online learning – are we selling the open and distance education message short?

This article questions the terminology being used to promote an institution’s programs. The author is particularly concerned that focusing on the term ‘online learning’ does a disservice to the special aspects of open and distance education. He argues it is necessary to pay close attention to the different needs of off-campus or distance learners, which can be lost in a discussion of the merits of online versus campus education. But above all, Todhunter is concerned that a focus on ‘online learning’ will put off many who are potential learners, whereas the terms ‘open’ and ‘distance’ will not only be be more appealing to some students, but may require different policies and strategies than a focus on ‘online’ learning.

Students embarking on graduate theses involving online learning, e-learning, distance education or open learning will benefit from reading this article when it comes to clearly defining what they are researching.

Comments

First, an explanation of why I have taken the time to ‘abstract’ these papers. This is not an ‘open access’ journal; you require a subscription from Taylor and Francis Group publications at nearly $40 an article. So pray that you have access to a good library, or you need to be sure that the article will be worth it to you. I have complained several times to Distance Education about a journal on open and distance education not being open access, but this is the policy of ODLAA (the Open and Distance Learning Association of Australia).

Second, some of the individual articles are well worth reading, depending on your interest. From reading the journal I picked up the following points (these are my interpretations, not necessarily the author’s):

  • good pedagogy is more important than mode of delivery (Hockridge) – further evidence for my law of equal substitution (i.e. most of what applies to good teaching in classrooms also applies to online education, and vice versa. Most things that can be taught in class can also be taught online, so we need to focus on the exceptions, not the rule.)
  • we need to do far more research and development on online assessment methods (Earl)
  • we are underusing learners’ life experiences in the design of distance courses (especially important for adult learners) (Watson)
  • institutions need better policies for casual/adjunct/contract instructors, and need to pay particular attention to professional development for this increasingly important human resource in higher education (Higgins and Harreveld)
  • even amongst the supporters of OERs, actual use, and especially secondary use, of OERs is still minimal (Murphy) – how long does maturation have to take?
  • studies of drop-out that focus on the demographics of incoming students are pretty useless. These are your students: find ways to help them succeed – don’t screen them out just because they are a higher risk, especially if you are an open institution (Yasmin)
  • open and distance learning are not necessarily the same as online learning; institutions need to be clear about markets and values as well as about mode of delivery. (Todhunter)

However, I do feel for journal editors who have to try to pick the best papers and at the same time try to find a common theme. The theme and the questions set out for this edition are only partly addressed in these papers, but nevertheless the articles are well worth reading. It’s just a pity they are so inaccessible.

Why are some public universities charging more for online courses?

Listen with webReader
Overheads constitute almost 50% of campus-based operational costs

Overheads constitute almost 50% of campus-based operational costs

Haynie, D. (2103) U.S. News Data: Online Education Isn’t Always Cheap, August 28

What

This report caught my eye: A 2013 survey conducted by the American Association of State Colleges and Universities and the Learning House, which will be published this October and involved 400 public universities, concluded that :

more than 60 percent charged the same tuition for face-to-face courses as they charged for online courses. Thirty-six percent of the schools charged more for online tuition.

This was also backed up by a USNews survey of 300 public universities. 

The average per credit, in-state cost for an online bachelor’s program is $277, compared with $243 per credit at brick-and-mortar schools.

I’m also aware that some Canadian institutions also have a premium fee or an additional charge for the online version of a face-to-face credit course.

Why

According to Susan Aldridge, a senior fellow at AASCU:

The courses cost more to develop, take more time to develop and take more time for the faculty to teach. In order for students to succeed in these online courses, 24/7 technical support, reference librarians, writing labs, automated degree plans and tutoring need to be available.

And Ray Schroeder adds:

schools often have to train their faculty to teach effectively on online platforms – an expensive, ongoing endeavor

Really?

Online learning costs more than classroom teaching? Not in my experience. Once again, it depends: on how you cost programs, on how you design your online courses, and how you deliver your face-to-face classes.

John Ebersole, president of Excelsior College, believes that online education is cheaper for colleges to provide because they don’t have to invest in creating or maintaining facilities. Those savings, he said, should outweigh the cost of any initial investment in technology.

If my online students aren’t going to take advantage of the cafeteria, going to the student union, participating in the extracurricular activities and we don’t have the building costs, why isn’t it cheaper?

In other words, the overheads are lower for online teaching than they are for classroom teaching – and since overheads constitute almost 50% of operational costs on a campus-based university, this is a significant factor.

And Ray says:

Most faculty members come prepared to teach face-to-face. They need substantial training and support in order to teach effectively. It’s not a one-time training.

Sorry, Ray, the first part of your statement is the problem. Many faculty do NOT come properly prepared to teach face-to-face. In fact what we are doing is expecting a lower standard for face-to-face teaching by not preparing them properly, while having to do it for online instructors.

In essence, online learning changes the cost structure of teaching. It does need more start-up investment and so there is a cost of change. If you have small online class numbers, it probably will cost more to go online. However, if you have too many students to accommodate on campus, going online will reduce your costs per student and there are some economies of scale as online enrollments increase. This is why it’s important for institutions to track the costs not only of going online, but the real cost of their campus-based classes, including overheads and faculty development (or lack of it).

Lastly, even if online learning is more expensive for the institution (which it needn’t be) there is an equity issue here. Why should students pay more for the same end-product just because it is packaged differently?

What do you think?

Does your institution charge more for online credit courses? If so, why?

Book review: OERs and Change in Higher Education: Reflections from Practice

Listen with webReader

Glennie, J. et al. (2012) Perspectives on Open and Distance Learning: OERs and Change in Higher Education: Reflections from Practice Vancouver BC/Paris: Commonwealth of Learning/UNESCO. The book is available for free downloading.

From the publisher’s web site:

Although OER activities are taking place globally, most large and well funded projects have been in North America and Europe. As a result, little is known about important questions such as how the more acute levels of resource constraint typical of developing countries impact on demand for OER and on their reuse. The case studies and reflections in this book cover OER practice and policy in a diverse range of contexts, with a strong focus on events in developing countries. However, the focus on experiences from the developing world is not exclusive, as valuable “generic lessons” applicable also to developing countries can be drawn from research in the more developed countries.

Review of the book

Hammer, S. (2013) Review of Perspectives on Open and Distance Learning: OERs and Change in Higher Education: Reflections from Practice British Journal of Educational Technology, Vol.44, no. 2

At the end of her review, Dr. Hammer states:

A key strength of this book is the breadth of coverage of issues that are relevant to OER combined with the particular challenges and opportunities that their use presents to poorer developing nations. I would recommend it to any educator interested in finding out more about this amazing movement and even starting to put these ideas into practice