August 14, 2018

Why is innovation in teaching in HE so difficult? 4. Integrating online and distance learning into the mainstream

Blended learning: what makes it innovative? Image: Erasmus+

This is the fourth and final post in this series. The previous three were:

Is it really so difficult?

A strong case could be made that at least in North America, higher education systems have been very successful in innovation. For instance, over the last 15 years, online learning has become widespread in most universities and colleges.

In the USA, one in three students now takes at least one distance education/online course for credit (Seaman et al., 2018). Although campus-based enrolments have been static or declining in the USA over the last few years, fully online enrolments have grown by about 5% over the last four years. 

In Canada, online learning in credit based courses has increased from around 5% of all enrolments in 2000 to around 15% of all enrolments in 2017. For the last four years, online enrolments have been increasing at a annual rate of between 12-16% in Canada, and nearly all universities and colleges in Canada now offer at least some fully online courses (Bates et al., 2017). 

However, that is one area where Canada differs from the USA. In the USA, online education is concentrated in a much smaller proportion of institutions in the USA than in Canada. In the USA, 235 institutions command 47% (2,985,347) of the student distance enrolments, but represent only 5% of all higher education enrolments in the USA (Seaman et al. 2018). Basically, some institutions, such as the University of Southern New Hampshire and Arizona State University, have become expert in scaling up online learning to a position where it has become large-scale and self-sustainable.

Then there are MOOCs. Many universities around the world are now offering MOOCs, with over 20 million enrolments a year. There may be criticism about completion rates and lack of accepted qualifications, but nevertheless, even – or especially – the elite universities have jumped on the MOOC bandwagon.

Also, Contact North’s project, Pockets of Innovation, with nearly 200 case studies, has identified that there are many individual instructors in colleges and universities adopting innovative uses of technology in their teaching, mostly independent of any institutional strategy.

However, probably the greatest impact of online learning on teaching in higher education is just getting started and that is the integration of online learning with classroom teaching, in the form of blended or hybrid learning. Bates et al. (2017) found that almost three quarters of institutions in Canada reported that this type of teaching was occurring in their institution. However, two thirds of the institutions reported that fewer than 10% of courses are in this format. In other words, integrated online learning is wide but not yet deep.

And this is where perhaps the biggest challenge of successful innovation lies: ensuring the high quality integration of online and classroom teaching. But we shall see that there are also concerns about how well campus-based institutions with no prior history of credit-based distance education have moved to fully online courses and programs as well.

The challenge of moving from a single mode to a dual mode institution

The most recent issue of the journal Distance Education, edited by Mays, Combrink and Aluko (2018) is a special edition dedicated to the theme of dual-mode provision, and in particular how previously single mode (i.e. solely campus-based) institutions are responding to the particular demands of distance education provision, and whether the quality and effectiveness of such provision is at risk. The editors of this edition believe:

such a decision will necessarily call for the revisiting of an institution’s assumptions about how people learn, how staff should work and how resources should be allocated and what policy changes are needed if quality is to be maintained or enhanced and the offerings sustained.

The articles in this special edition raise a number of questions such as:

  • is the blurring of the boundaries between on-campus and distance learning a good thing?
  • does the concept of distance education remain relevant?
  • are established models of distance education sufficient to inform the design, development and delivery of new kinds of provision, or are new models emerging (or needed)?

In particular, the editors are concerned that:

  • there is a real danger that in the convergence of modes of provision the unique quality concerns of distance provision, regarding, for example, the issues of access, success and cost, and the implications for how people learn and work, may be lost.

Interestingly, the special edition then looks at a series of case studies of the move from single to dual mode not drawn from North America or Europe, but from sub-Saharan Africa, where the motivation to move into distance learning has been driven mainly by changes in demand patterns (too many potential students; not enough institutions).

Application of an innovation adoption framework

Of these case studies, by far the most interesting is the article by Kanwar et. al, of the Commonwealth of Learning, which applies Wisdom et al.’s (2014) innovation adoption framework to provide a qualitative meta-review of barriers to adoption of open and distance learning (ODL) in conventional higher education institutes in Cameroon, Kenya and Rwanda. 

The framework has four key elements (which build on Everett Rogers’ earlier work on the diffusion of innovation):

  • external environment, e.g. national policies and funding, infrastructure/external physical environment
  • organisation of the adopting institution, e.g. institutional policies, organisational structure, leadership
  • nature of the innovation, e.g. complexity, cost, technology 
  • individuals, e.g. skills, perceptions, motivation, value systems of staff and clients affected by the innovation.

Kanwar et al. then used this framework to analyse the content of existing reviews of the adoption of ODL in the three countries. The findings are too detailed and complex to review here (the results varied between the three countries), but the study clearly identified some of the key barriers to adoption in each of the three countries. I was in fact thrilled to see an evidence-based theoretical model used to evaluate innovation.

More importantly, the study resulted in nine recommendations for successful implementation of ODL within campus-based institutions:

Government

  • develop national level policies and funding to encourage the adoption of ODL
  • establish national-level quality assurance mechanisms, equally for on-campus and distance programs
  • strengthen national-level IT infrastructure

Institutions

  • create institutional policies and clear implementation plans for promoting and supporting ODL
  • establish a centralised and autonomous ODL structure
  • develop a clear costing model for ODL and establish secure forms of funding/business models
  • build staff capacity and provide incentives to faculty to engage in ODL
  • promote research into the effectiveness and outcomes of ODL
  • ensure equivalency in the status and qualifications of ODL students

Comment

It would be a mistake to ignore this publication because the cases are drawn primarily from sub-Saharan Africa. Many of the issues addressed in these articles will resonate with many working in this field in North America and Europe.

I think the editors are right to be concerned about how well ‘conventional’ institutions are handling the adoption of distance and online learning. For many faculty, moving online is merely a question of transferring their classroom lectures to a web conference.

I was at a Canadian university recently where the design of a ‘blended’ executive MBA was being discussed. The ‘plan’ was to make one of the three weekly lectures in each course available instead by a 90 minute synchronous web conference. One professor insisted that all students had to watch the lecture at the same time so they could discuss it afterwards. No consideration was given to either the context of the students (working businessmen with a busy schedule and family) or to the pedagogy or research on video lectures. Even worse, the faculty were not listening to advice from the excellent specialists from the university’s Centre for Teaching and Learning.

At another Canadian university which has been running excellent distance education program for years through Extension Services, there is no plan or strategy for e-learning on campus, other than a proposal to distribute the specialist instructional design staff from Extension to the campus-based academic departments (which wouldn’t work as there are not enough specialists to go round each faculty). This also ignores the fact that these specialists are needed to run Extension’s own very successful non-credit programs, which bring money into the university.

So looking down the list of recommendations suggested by Kanwar et al., I can immediately think of at least a dozen Canadian universities for which most of these recommendations would be highly relevant.

I would differ on just a couple of points. There has been a long tradition of dual-mode institutions in North America, especially in universities with a state- or province-wide remit, at least in their early days. In Canada, Queen’s and Guelph Universities in Ontario, Memorial University in Newfoundland, the University of Saskatchewan, Laval University in Québec, and the University of British Columbia are all examples of mainly campus-based institutions with very successful distance programs. The distance education programs were the first to adopt online learning, and gradually, some of the best practices from distance education have been incorporated into blended and hybrid courses.

However, even in these universities, the move to more integrated online and face-to-face teaching faces challenges. UBC for instance did move its distance education staff from Continuing Studies to join a strengthened Centre for Teaching, Learning and Technology that also included faculty development. Other institutions have still to make that move in a strategic and careful manner. And the big issue is how do you scale from supporting online courses for 15% of the students to supporting blended learning for all students?

The real issue lies with faculty and especially departments moving to blended or hybrid learning that do not understand the need for learning design or the needs of students who are not on campus all the time. The integration of online and campus-based learning will often highlight the inadequacy of prior campus-based teaching methods. There is much that campus-based faculty can learn from  distance education, in terms of more effective teaching.

At the same time, I don’t think distance educators have all the answers. The Pockets of Innovation have plenty of examples of campus-based faculty thinking up innovative ways to integrate online learning and new technologies into campus-based teaching. My experience in designing online courses was that the best ideas usually came from a highly expert faculty member with a truly deep understanding of the subject matter (see my previous post on VR in interactive molecular mechanics for a good example). I believe that we will need new models for designing blended and hybrid courses, even though distance education has some sound principles that can guide such design.

So in conclusion, innovation of itself is not sufficient: it has to be effective innovation that leads to better outcomes, in terms of access, flexibility, and/or learning effectiveness. Innovation is unlikely to be effective if it merely moves poor classroom teaching online, which is why innovation will remain difficult in higher education.

Over to you

Do you have examples of poor practice in moving to offer distance education courses for the first time, or attempts at integrating online and classroom teaching? 

Even better, do you have examples of where this has been done successfully? What are the lessons you have learned from this?

References

Bates, T. (ed.) (2017) Tracking Online and Distance Education in Canadian Universities and Colleges: 2017 Vancouver BC: The National Survey of Online and Distance Education in Canadian Post-Secondary Education.

Kanwar, A. et al., (2018) Opportunities and challenges for campus-based universities in Africa to translate into dual-mode delivery, Distance Education, Vol. 39. No. 2, pp. 140-158

Mays, T. et al. (2018) Deconstructing dual-mode provision in a digital era, Distance Education, Vol. 39. No. 2

Seaman, J.E., Allen, I.E., and Seaman, J. (2018) Grade Increase: Tracking Distance Education in the United StatesWellesley MA: The Babson Survey Research Group

Wisdom, J. et al. (2014) Innovation adoption: a review of theories and constructs, Administration and Policy in Mental Health and Mental Health Sciences Research, Vol. 41, pp. 480-502

Book review: Open and Distance Education in Australia, Europe and the Americas

Qayyum, A. and Zawacki-Richter, O. (eds.) Open and Distance Education in Australia, Europe and the Americas: National Perspectives in a Digital Age Singapore: Springer, US$24+

Why this book?

This book is the first of two volumes aimed at describing how open and distance education (ODE) is evolving to reflect the needs and circumstance of the national higher education systems in various countries. A second goal is to compare how DE is organized and structured in various countries.

What does the book cover?

This first volume covers Australia, Europe and the Americas; the second book (still to come) covers Asia, Africa and the Middle East (including Russia and Turkey).

Who wrote it?

This is a well-edited book, with individual chapters written by experts within each country, following a roughly consistent structure in terms of topics. There is a main chapter for each country, with a useful second opinion from another country expert in terms of a commentary on the main chapter, as follows:

  • Introduction (ODE in a Digital Age): Qayyum and Zawacki-Richter
  • Australia: Colin Latchem (commentary by Som Naidu)
  • Brazil: Fred Litto (commentary by Maria Renata da Cruz Duran and Adnan Qayyum)
  • Canada: Tony Bates (commentary by Terry Anderson)
  • Germany: Ulrich Bernath and Joachim Stöter (commentary by Burkhard Lehmann)
  • United Kingdom: Anne Gaskell (commentary by Alan Tait)
  • United States of America: Michael Beaudoin (commentary by Gary Miller)
  • Conclusions: Qayyum and Zawacki-Richter

What’s in it?

There is some variation between the chapters, reflecting some of the differences between different countries, but most chapters have the following structure:

  • Context: most chapters start with a section that provides the wider context in which ODE operates within a country, either in terms of history or a brief description of the current higher education system as a whole. This sometimes includes how DE is funded (or not funded) by governments.
  • Enrolments and growth: each chapter attempts (heroically in some cases) to estimate just how many distance education students there are within the country and the rate of growth. What is noticeable here is how much variation there is in the accuracy or reliability of these estimates between different countries, partly because of the blurring of definitions between online and blended learning, but partly because in some countries, no-one seems to be counting.
  • Quality assurance/quality control: this describes both the regulatory framework for HE within each country and how that is applied to ODE.
  • Descriptions of specific ODE institutions: these sections describe those specialized institutions that play a major role in ODE within their respective countries.
  • OER and MOOCs. Most chapters discuss the use of open educational resources and MOOCs in their country.
  • The relationship between public and private provision of ODE. This is very useful as the relationship varies considerably between different countries.
  • The future of ODE within each country: this section looks at both challenges and opportunities.

In addition, Qayyum and Zawicki-Richter provide an excellent concluding chapter, that compares the different countries in terms of:

  • size and growth of ODE: ODE enrolments constitute between at least 10-20% of all HE enrolments in Australia, Brazil, Canada and the USA. In the UK and Germany, though, the proportions are likely to be less than 10%;
  • providers of DE: one reason reliable data collection has been difficult is because of the growth in different types of institutions providing DE: specialized ODE providers have in general increased their numbers; more campus-based institutions have become providers of ODE; and private institutions offering ODE have grown. However, this varies considerably from country to country. In the UK, for instance, ODE enrolments have been dropping at the UKOU, but possibly increasing from campus-based providers. In the USA, enrolments from the for-profit ODE providers have been dropping but increasing in the private and public on-campus institutions. What is clear is the impact on ODE enrolments of government policies regarding funding and tuition fees;
  • online vs other forms of DE: again, this differs between countries (and probably even more so in the countries to be covered in the next book). In Australia, Canada, the USA and the UK, ODE is nearly synonymous with online learning; Brazil has ‘leapfrogged’ to mobile learning;
  • the role of government: too complex to summarise here: read the chapter!
  • the function of ODE: ODE appears to play three major functions in HE systems: increasing access; providing greater flexibility to those with access; and ‘abetting in the larger digital transformation of HE’;
  • trends and future challenges: ODE on a macro level is being affected by two factors: the global growth in demand for HE; and the digital revolution. Surprisingly, though, it is less affected by globalization: ‘ODE seems to function mainly, though not wholly, within the nation state’ – except for MOOCs. This chapter has a very good discussion of these issues, particularly the differences between education as a public or private good, and ODE’s role in each.

My comments

The book sets out clearly the extent and importance of ODE in higher education. A careful reading will also indicate the importance of government and institutional policies in supporting or restricting ODE.

This and the second book in this series therefore should be required reading in any post-graduate education program. It should also be required reading by policy analysts in Ministries of Advanced (or Higher) Education. I would also recommend it to Boards of Governors and Provosts/VP Academic in any post-secondary institution. 

I look forward with impatience to reading the second volume, which for me will be even more valuable as I know so little about ODE in many of the countries covered in the second book.

If I have any negative comments, it is about what is not in the book. I think it is a pity that there is no chapter on France, Mexico or Argentina, all of which are very large countries with substantial and uniquely different distance education provision. And of course it is solely about formal post-secondary education. Other books are needed to cover international distance education in the k-12 and corporate sectors.

Also, this book will easily become outdated, given the rapid developments in ODE around the world. It took over two years from the time I was approached to write the chapter and the book’s publication. In this period, the first national survey of online learning in Canadian post-secondary education was published, the results of which had to be hastily accommodated in the last proofs of the book.

Furthermore, the book is an open publication, and is free to download, licensed as open access under a Creative Commons Attribution 4.0 International License. However, it is not expensive to buy a hard copy, and I hope if you have an an interest in open and distance education you will make this a standard book on your shelves – after you have read it!

(Note: in an earlier version of this post I incorrectly stated that it could not be downloaded for free. My apologies).

Why is innovation in teaching in higher education so difficult? 1. The accreditation agencies?

Southern New Hampshire University’s College for America is considered a successful innovation, according to the Christensen Institute

Horn, M. and Dunagan, A. (2018) Innovation and Quality Assurance in Higher Education San Francisco CA: The Christensen Institute

Revisiting an old question

A couple of things recently have led me to reflect once again on this question. There are some obvious reasons for a lack of innovation in teaching in higher education, such as:

  • lack of pedagogical training for post-secondary instructors,
  • the privileging of research over teaching,
  • lack of rewards for good teaching (or lack of punishment for poor teaching)
  • faculty fear of technology,

but there are other, perhaps more subtle, factors that make innovation and change so damned difficult in universities and colleges.

One factor is suggested in the report from the Christensen Institute, which puts the blame squarely on accreditation agencies. I will look at this claim in this post.

However, responses to my recent post on learning management systems, where I suggested that the time has come to move on to other tools for online learning, also suggest other reasons why even online learning is becoming increasingly resistant to change. I will examine this issue – which I think is much more significant – in my next post.

The ‘dead hand’ of accreditation agencies

The more insidious failure of accreditation is the stifling effect it has on innovation at existing institutions.

This is the conclusion from a report from the Christensen Institute (yes, that Christensen, the disruptive one) based on four case studies (yes, just four).

It looked at attempts by the following four institutions in setting up an online operation separate from the main, campus-based institution’s teaching model:

  • Bellevue University’s (Nebraska) Flexxive Program
  • Tiffin University’s (Ohio) Ivy Bridge Program
  • Southern New Hampshire University’s College for America
  • General Assembly’s coding boot camps

The first two innovations failed; the second two succeeded. The difference, according to the report, was the role of the respective accrediting agencies.

The authors argue:

Innovations aimed at redefining a college or university’s value proposition must be insulated from its existing business model or else it will conform to the inputs of the existing business model rather than create a new one….Creating an autonomous unit is critical for a college to launch an innovation aimed at dramatically transforming its value proposition.

Depending on the nature of the innovation, a college or university must work closely with its accreditor to ensure that the new practice is consistent with the accreditor’s quality standards. As a result, accreditation plays a major role in the innovation process for most colleges and universities.

Accreditation as it currently stands is inconsistent, both between accreditors, and between the same accreditor at different points in time. Standards of accreditation vary between accreditors, but their interpretation varies to a larger degree—even between different accrediting teams looking at the same institution. This creates uncertainties for institutional leaders and creates untenable risks for many schools with limited resources that are considering whether to bring innovative programs forward. ….. Institutions that are able to innovate are those blessed by geography—a cooperative, forward-thinking regional accreditor— as well as finances.

… a process that is so subject to individual interpretation and has a track record of inconsistently applying rules and standards cannot be a foundation for regulation supportive of innovation. As countless scholars have shown, investment in innovation does not thrive in climates of uncertainty.

Is it true?

This is a pretty damning condemnation of American accreditation agencies, and I suggest needs to be taken with a grain of salt. I myself have argued that professional accreditation agencies, such as the Professional Engineers of Ontario, certainly stifle innovation when they refuse to accept any qualifications taken through distance education. But university and college accreditation?

There are several reasons why I think the Christensen Institute’s conclusions are too strong:

  • online learning has been expanding rapidly in the USA over the last 10 years, where at least one in three students are taking an online course, a rate of growth much faster than campus-based enrolments, yet the accreditation agencies have done little to prevent this fairly major innovation in teaching; 
  • I challenge the assumption – which is at the core of the Christensen philosophy – that innovation can take place only if it is insulated from an organization’s existing business model. Sure, by definition this may be true of disruptive innovation in business, but nevertheless there has been considerable innovation in terms of introducing online and more recently blended learning in higher education, without disrupting the current business model of universities and colleges;
  • the university accreditation process in the USA is unique, lacks rationality generally in terms of its relationship between government, geography, and institutional governance, and is indeed often inconsistent and contradictory, not only with regard to innovation but often with regard to traditional programs. But yes, it has slowed down – but not prevented – innovation through online learning.

In other words, while no doubt US higher education accreditation agencies may inhibit innovation to some extent, especially with regard to radically new institutions (but perhaps based on a reasonable assessment of possible risk to fee-paying students), this report is too much like a theory trying to find evidence to support it, rather than a systematic study of what not only inhibits but also what enables innovation in higher education. The barriers to innovation in higher education are more complex than just being the fault of the nasty accreditation agencies. More on this in the next post.

Book review: Open and Distance Non-formal Education in Developing Countries

A mobile school for Delhi street children run by Butterflies. Click to see video

Latchem, C. (2018) Open and Distance Non-formal Education in Developing Countries Springer: Singapore

The author

I was about to review this book when I was informed of the death of Colin Latchem, its author.

Colin was an Australian consultant, researcher and writer in the field of open and distance learning.  In the 1970s, he was a pioneer in the UK in the development of educational television and learning resources for universities.

He emigrated to Australia in 1982 to become head of the Teaching Learning Group at Curtin University, Perth, a centre responsible for academic staff development, educational technology and open and distance learning. Over the years he became the ‘go-to’ person about open and distance education in South East Asia. He received the Charles A Wedemeyer award in 2002 for best book of the year on open and distance education. He was also co-editor of the SpringerBriefs series on Open and Distance Education. He was formerly the Asia-Pacific Corresponding Editor of The British Journal of Educational Technology.

Colin was a good friend and colleague whom I have known for over 40 years. I cannot think of a more appropriate way to celebrate a true scholar and gentleman than to review his final work.

Definition of open and distance non-formal education (ODL NFE)

Latchem does not provide a precise definition of non-formal education, but distinguishes non-formal learning from informal learning (the spontaneous, incidental acquisition of knowledge) and formal learning provided by schools, colleges and universities. Non-formal learning sits somewhere in between, concerned with providing lifelong learning in support of social equality, employment and development for those denied formal education. It may be provided through NGOs, international or government agencies, employers or social organisations such as community groups.

In open and distance education most of the teaching is conducted by some provider removed in time and space from the learner, using content and approaches that are openly accessible, enabling learners to learn individually or collaboratively at the time and place of their choosing.

The importance of open and distance education for non-formal education

 Some of the figures Latchem provides about the need for non-formal education are staggering: 

  • 263 million children and youth did not have access to schools in 2014
  • 130 million girls are denied the right to formal education, and are four times more likely to be denied education than boys of the same socio-economic group
  • 758 million adults aged 15 years and older remain illiterate, of which two-thirds are women 
  • there are 60 million refugees or displaced persons without access to formal education
  • it would take an extra US$40 billion to provide 12 years of education for all in the developing world, but international aid today is 4% lower than it was in 2010.

Other groups outside the formal education system in developing countries include people with disabilities and people imprisoned. It is of course still the poorest socio-economic groups who have the least access to formal education in developing countries, despite often heroic efforts by national governments.

Latchem argues that conventional face-to-face methods can never meet the scale and extent of the knowledge and skills building and social and behavioural change needed to meet the United Nations’ Millenium Development Goals. Open and distance education non-formal education (ODL NFE) is the only way to meet these needs until formal educational provision becomes globally available to all, and even then ODL NFE will still be needed on a large scale.

However, Latchem claims that there has been little prior research into the effectiveness of ODL NFE in developing countries. What little prior research that has been done indicates that previous attempts to use open and distance learning for non-formal education in developing countries were piecemeal and ineffective, mainly consisting of short-term pilots lacking sustainable funding.  

Latchem concluded that a review of current practice and progress in this field was long overdue and hence the central concern of the book is about identifying ways in which open and flexible forms of lifelong learning have increased social equality, employment and development for those denied formal education.

The structure of the book

There are four parts to the book:

  1. Background to the study, which examines the Global Development Agenda, and introduces the reader to prior research, and the main elements of ODL NFE.
  2. A fairly brief description of the main technologies and media currently in use in ODL NFE, including radio, television, mobile learning, OERs and MOOCs, telecentres, and traditional and performing arts.
  3. A more extensive review of areas in which ODL NFE has been mostly successfully used. These include:
    • out-of-school children and youth
    • adult literacy, ESL
    • gender equity
    • disabled, refugees, prisoners
    • health care, safe water, sanitation and hygiene
    • agriculture and agribusiness
    • small and medium-sized enterprises
    • education for sustainable development
  4. A conclusion, including actions needed

My main takeaways

Firstly, the size of the challenge in providing education for all. I agree with Latchem that although the long-term goal should be formal education for all, in the short-term this will be impossible for many years in many developing countries, and that non-formal education will continue to be critically important in helping to fill the gap, and that open and distance learning is a valuable, cost-effective means to provide this. (It is also cost-effective means to provide formal education, as well, but that is another book).

Second, though, I was blown away by the many cases Latchem provides of successful ODL NFE projects. The book is filled with over 180 cases and urls to video links which demonstrate the applications. I was particularly impressed by the extent and value of telecentres, and the criteria needed for them to succeed. There are lessons here for developed as well as developing countries.

Third, while cost and access remain a major barrier, I was impressed by the extent to which the Internet and ICTs (particularly mobile learning) are being successfully used in many developing countries. I was also impressed with the use of more traditional media, such as puppets, theatre, song and dance, highlighting the importance of cultural adaptation to local needs. Again I believe there are lessons here for developed as well as developing countries.

Nevertheless, while these success stories are encouraging, there are often systemic difficulties that hinder the implementation of ODL NFE. Latchem identifies the following:

  • over-dependence on international aid agencies/NGOs
  • lack of sustainability due to overuse of short-term, small scale pilots and insufficient funding
  • lack of learning pathways from informal to non-formal to formal education
  • the need for a systematic approach/a national strategy for non-formal education
  • lack of reliable broadband connection in rural areas where NFE is most needed
  • lack of content in local languages
  • lack of research and evaluation of projects in terms of outcomes.

Latchem then ends with a set of nine action steps that are needed to advance the ODL NFE agenda.

In summary

This book benefits enormously from being written by a single author, rather than a series of articles by different writers. This provides the book with a coherent and consistent message.

I cannot say how thrilled I was to see so many wonderful projects attempting under great difficulty to make the world a better place. Most of these were firmly community-based, and locally designed and maintained, if often with some international assistance. It is one of the most optimistic books I have read for a long while.

It also highlights the naïvity and wrong-headedness of many Western approaches to the use of technology in developing countries, such as believing the importation of American MOOCs (or whatever is the latest technology) is a sustainable solution to education for all. There is a role for MOOCs, but are best developed locally in local languages, for instance, and more importantly, embedded in a local organisation and infrastructure that makes the material likely to be used effectively.

Some of the early content will be familiar to most readers of this blog, but the real target for this book are policy-makers in developing countries trying to tackle the challenge of education for all. This book provides powerful evidence of the role that open and distance education non-formal education can play in making education for all a reality. This is a fitting end to a wonderful career – thank you, Colin.

Learning analytics, student satisfaction, and student performance at the UK Open University

There is very little correlation between student satisfaction and student performance. Image: Bart Rienties. Click on image to see the video.

Rienties, B. and Toetenel, L. (2016) The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules, Computers in Human Behaviour, Vol. 60, pp.333-341

Li, N. et al. (2017) Online learning experiences of new versus continuing learners: a large-scale replication study, Assessment and Evaluation in Higher Education, Vol. 42, No. 4, pp.657-672

It’s never too late to learn

It’s been a hectic month with two trips from Vancouver to Ontario and back and one to the UK and back, a total of four keynotes, two panel sessions and two one day consultancies. By the time I got to the end of the month’s travels, I had learned so much that at a conference in Toronto I had to go to my room and lie down  – I just couldn’t take any more!

At my age, it takes time to process all this new information, but I will try to summarise the main points of what I learned in the next three posts.

Learning analytics at the Open University

The Open University, with over 100,000 students and more than 1,000 courses (modules), and most of its teaching online in one form or another, is an ideal context for the application of learning analytics. Fortunately the OU has some of the world leaders in this field. 

At the conference on STEM teaching at the Open University that I attended as the opening keynote, the closing keynote was given by Bart Rienties, Professor of Learning Analytics at the Institute of Educational Technology at the UK Open University. Rienties and his team linked 151 modules (courses) and 111,256 students with students’ behaviour, satisfaction and performance at the Open University UK, using multiple regression models. 

His whole presentation (40 minutes, including questions) can be accessed online, and is well worth viewing, as it provides a clear summary of the results published in the two detailed papers listed above. As always, if you find my summary of results below of interest or challenging, I strongly recommend you view Bart’s video first, then read the two articles in more detail. Here’s what I took away.

There is little correlation between student course evaluations and student performance

This result is a bit of a zinger. The core dependent variable used was academic retention (the number of learners who completed and passed the module relative to the number of learners who registered for each module). As Rientes and Toetenel (p.340) comment, almost as an aside, 

it is remarkable that learner satisfaction and academic retention were not even mildly related to each other….Our findings seem to indicate that students may not always be the best judge of their own learning experience and what helps them in achieving the best outcome.’

The design of the course matters

One of the big challenges in online and blended learning is getting subject matter experts to recognise the importance of what the Open University calls ‘learning design.’ 

Conole (2012, p121) describes learning design as:

a methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies. LD is focussed on ‘what students do’ as part of their learning, rather than the ‘teaching’ which is focussed on the content that will be delivered.

Thus learning design is more than just instructional design.

However, Rienties at al. comment that ‘only a few studies have investigated how educators in practice are actually planning and designing their courses and whether this is then implemented as intended in the design phase.’ 

The OU has done a good job in breaking down some of the elements of learning design. The OU has mapped the elements of learning design in nearly 200 different courses. The elements of this mapping can be seen below (Rientes and Toetenal, 2016, p.335):

Rientes and Toetenel then analysed the correlations between each of these learning design elements against both learner satisfaction and learner performance. What they found is that what OU students liked did not match with learner performance. For instance, students were most satisfied with ‘assimilative’ activities, which are primarily content focused, and disliked communication activities, which are primarily social activities. However, better student retention was most strongly associated with communication activities, and overall, with the quality of the learning design.

Rientes and Toetenel conclude:

although more than 80% of learners were satisfied with their learning experience, learning does not always need to be a nice, pleasant experience. Learning can be hard and difficult at times, and making mistakes, persistence, receiving good feedback and support are important factors for continued learning….

An exclusive focus on learner satisfaction might distract institutions from understanding the impact of LD on learning experiences and academic retention. If our findings are replicated in other contexts, a crucial debate with academics, students and managers needs to develop whether universities should focus on happy students and customers, or whether universities should design learning activities that stretch learners to their maximum abilities and ensuring that they eventually pass the module. Where possible, appropriate communication tasks that align with the learning objectives of the course may seem to be a way forward to enhance academic retention.

Be careful what you measure

As Rientes and Toetenel put it:

Simple LA metrics (e.g., number of clicks, number of downloads) may actually hamper the advancement of LA research. For example, using a longitudinal data analysis of over 120 variables from three different VLE/LMS systems and a range of motivational, emotions and learning styles indicators, Tempelaar et al. (2015) found that most of the 40 proxies of simple” VLE LA metrics provided limited insights into the complexity of learning dynamics over time. On average, these clicking behaviour proxies were only able to explain around 10% of variation in academic performance.

In contrast, learning motivations, emotions (attitudes), and learners’ activities during continuous assessments (behaviour) significantly improved explained variance (up to 50%) and could provide an opportunity for teachers to help at-risk learners at a relatively early stage of their university studies.

My conclusions

Student feedback on the quality of a course is really important but it is more useful as a conversation between students and instructors/designers than as a quantitative ranking of the quality of a course.  In fact using learner satisfaction as a way to rank teaching is highly misleading. Learner satisfaction encompasses a very wide range of factors as well as the teaching of a particular course. It is possible to imagine a highly effective course where teaching in a transmissive or assimilative manner is minimal, but student activities are wide, varied and relevant to the development of significant learning outcomes. Students, at least initially, may not like this because this may be a new experience for them, and because they must take more responsibility for their learning. Thus good communication and explanation of why particular approaches to teaching have been chosen is essential (see my comment to a question on the video).

Perhaps though the biggest limitation of student satisfaction for assessing the quality of the teaching is the often very low response rates from students, limited evaluation questions due to standardization (the same questions irrespective of the nature of the course), and the poor quality of the student responses. This is no way to assess the quality of an individual teacher or a whole institution, yet far too many institutions and governments are building this into their evaluation of teachers/instructors and institutions.

I have been fairly skeptical of learning analytics up to now, because of the tendency to focus more on what is easily measurable (simple metrics) than on what students actually do qualitatively when they are learning. The focus on learning design variables in these studies is refreshing and important but so will be analysis of student learning habits.

Finally, this research provides quantitative evidence of the importance of learning design in online and distance teaching. Good design leads to better learning outcomes. Why then are we not applying this knowledge to the design of all university and college courses, and not just online courses? We need a shift in the power balance between university and college subject experts and learning designers resulting in the latter being treated as at least equals in the teaching process.

References

Conole, G. (2012). Designing for learning in an open world. Dordrecht: Springer

Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: learning analytics in a data-rich context. Computers in Human Behavior, 47, 157e167. http://dx.doi.org/10.1016/j.chb.2014.05.038.