June 21, 2018

Learning analytics, student satisfaction, and student performance at the UK Open University

There is very little correlation between student satisfaction and student performance. Image: Bart Rienties. Click on image to see the video.

Rienties, B. and Toetenel, L. (2016) The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules, Computers in Human Behaviour, Vol. 60, pp.333-341

Li, N. et al. (2017) Online learning experiences of new versus continuing learners: a large-scale replication study, Assessment and Evaluation in Higher Education, Vol. 42, No. 4, pp.657-672

It’s never too late to learn

It’s been a hectic month with two trips from Vancouver to Ontario and back and one to the UK and back, a total of four keynotes, two panel sessions and two one day consultancies. By the time I got to the end of the month’s travels, I had learned so much that at a conference in Toronto I had to go to my room and lie down  – I just couldn’t take any more!

At my age, it takes time to process all this new information, but I will try to summarise the main points of what I learned in the next three posts.

Learning analytics at the Open University

The Open University, with over 100,000 students and more than 1,000 courses (modules), and most of its teaching online in one form or another, is an ideal context for the application of learning analytics. Fortunately the OU has some of the world leaders in this field. 

At the conference on STEM teaching at the Open University that I attended as the opening keynote, the closing keynote was given by Bart Rienties, Professor of Learning Analytics at the Institute of Educational Technology at the UK Open University. Rienties and his team linked 151 modules (courses) and 111,256 students with students’ behaviour, satisfaction and performance at the Open University UK, using multiple regression models. 

His whole presentation (40 minutes, including questions) can be accessed online, and is well worth viewing, as it provides a clear summary of the results published in the two detailed papers listed above. As always, if you find my summary of results below of interest or challenging, I strongly recommend you view Bart’s video first, then read the two articles in more detail. Here’s what I took away.

There is little correlation between student course evaluations and student performance

This result is a bit of a zinger. The core dependent variable used was academic retention (the number of learners who completed and passed the module relative to the number of learners who registered for each module). As Rientes and Toetenel (p.340) comment, almost as an aside, 

it is remarkable that learner satisfaction and academic retention were not even mildly related to each other….Our findings seem to indicate that students may not always be the best judge of their own learning experience and what helps them in achieving the best outcome.’

The design of the course matters

One of the big challenges in online and blended learning is getting subject matter experts to recognise the importance of what the Open University calls ‘learning design.’ 

Conole (2012, p121) describes learning design as:

a methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies. LD is focussed on ‘what students do’ as part of their learning, rather than the ‘teaching’ which is focussed on the content that will be delivered.

Thus learning design is more than just instructional design.

However, Rienties at al. comment that ‘only a few studies have investigated how educators in practice are actually planning and designing their courses and whether this is then implemented as intended in the design phase.’ 

The OU has done a good job in breaking down some of the elements of learning design. The OU has mapped the elements of learning design in nearly 200 different courses. The elements of this mapping can be seen below (Rientes and Toetenal, 2016, p.335):

Rientes and Toetenel then analysed the correlations between each of these learning design elements against both learner satisfaction and learner performance. What they found is that what OU students liked did not match with learner performance. For instance, students were most satisfied with ‘assimilative’ activities, which are primarily content focused, and disliked communication activities, which are primarily social activities. However, better student retention was most strongly associated with communication activities, and overall, with the quality of the learning design.

Rientes and Toetenel conclude:

although more than 80% of learners were satisfied with their learning experience, learning does not always need to be a nice, pleasant experience. Learning can be hard and difficult at times, and making mistakes, persistence, receiving good feedback and support are important factors for continued learning….

An exclusive focus on learner satisfaction might distract institutions from understanding the impact of LD on learning experiences and academic retention. If our findings are replicated in other contexts, a crucial debate with academics, students and managers needs to develop whether universities should focus on happy students and customers, or whether universities should design learning activities that stretch learners to their maximum abilities and ensuring that they eventually pass the module. Where possible, appropriate communication tasks that align with the learning objectives of the course may seem to be a way forward to enhance academic retention.

Be careful what you measure

As Rientes and Toetenel put it:

Simple LA metrics (e.g., number of clicks, number of downloads) may actually hamper the advancement of LA research. For example, using a longitudinal data analysis of over 120 variables from three different VLE/LMS systems and a range of motivational, emotions and learning styles indicators, Tempelaar et al. (2015) found that most of the 40 proxies of simple” VLE LA metrics provided limited insights into the complexity of learning dynamics over time. On average, these clicking behaviour proxies were only able to explain around 10% of variation in academic performance.

In contrast, learning motivations, emotions (attitudes), and learners’ activities during continuous assessments (behaviour) significantly improved explained variance (up to 50%) and could provide an opportunity for teachers to help at-risk learners at a relatively early stage of their university studies.

My conclusions

Student feedback on the quality of a course is really important but it is more useful as a conversation between students and instructors/designers than as a quantitative ranking of the quality of a course.  In fact using learner satisfaction as a way to rank teaching is highly misleading. Learner satisfaction encompasses a very wide range of factors as well as the teaching of a particular course. It is possible to imagine a highly effective course where teaching in a transmissive or assimilative manner is minimal, but student activities are wide, varied and relevant to the development of significant learning outcomes. Students, at least initially, may not like this because this may be a new experience for them, and because they must take more responsibility for their learning. Thus good communication and explanation of why particular approaches to teaching have been chosen is essential (see my comment to a question on the video).

Perhaps though the biggest limitation of student satisfaction for assessing the quality of the teaching is the often very low response rates from students, limited evaluation questions due to standardization (the same questions irrespective of the nature of the course), and the poor quality of the student responses. This is no way to assess the quality of an individual teacher or a whole institution, yet far too many institutions and governments are building this into their evaluation of teachers/instructors and institutions.

I have been fairly skeptical of learning analytics up to now, because of the tendency to focus more on what is easily measurable (simple metrics) than on what students actually do qualitatively when they are learning. The focus on learning design variables in these studies is refreshing and important but so will be analysis of student learning habits.

Finally, this research provides quantitative evidence of the importance of learning design in online and distance teaching. Good design leads to better learning outcomes. Why then are we not applying this knowledge to the design of all university and college courses, and not just online courses? We need a shift in the power balance between university and college subject experts and learning designers resulting in the latter being treated as at least equals in the teaching process.

References

Conole, G. (2012). Designing for learning in an open world. Dordrecht: Springer

Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: learning analytics in a data-rich context. Computers in Human Behavior, 47, 157e167. http://dx.doi.org/10.1016/j.chb.2014.05.038.

 

Online learning and disruptive change at the UK Open University

The old Walton Hall on the OU campus in Milton keynes

Sturm und Strang

I’ve was in England last week,  attending the 7th eSTEeM conference at the Open University as the opening keynote speaker, only my second visit to the OU since I left nearly 30 years ago.

The Open University, described by several commentators as one of the most successful innovations in Britain since the Second World War, is currently going through an existential crisis, which culminated two weeks ago with the resignation of its Vice-Chancellor, Peter Horrocks, following a devastating vote of no confidence by faculty and staff.

The OU is facing enormous pressure, due mainly to the policies of the recent Conservative governments. Over the last six years, the government has treated the OU just the same as other, more traditional, universities in England and Wales. The government severely cut the OU’s operating budget requiring it to dramatically increase fees, and also made all part-time students (i.e. students not taking a full annual course load) ineligible for government-guaranteed, low interest loans. It also has required students at the OU, like all other students in England and Wales, to complete their bachelor studies within three years, compressing their time for study. It is expected to have a £20 million (CS$36 million) operating deficit this year and was proposing to save £100m from its £420m annual budget by cutting courses and staff.

Since the vast majority of its 200,000 students in 2012 were part-time, working adults without a first degree and who required the maximum flexibility in their studies, it’s hardly surprising that its student numbers have dropped by more than a third since 2012. At the same time it has invested heavily in FutureLearn, a MOOC-type platform which is still struggling to find a viable business model. The recent changes mean that the whole concept of open-ness and accessibility for OU students, and its unique position in the British higher education system, are under existential threat. 

To cap all this, the university itself recognises that it needs to fundamentally change its operational model. Like many other Open Universities, it has not changed fast enough to accommodate to the digital revolution in post-secondary teaching. It is burdened with a heavy legacy of a print-based design model and an expensive regional tutoring system, despite the recent elimination of all local face-to-face operations.

“We want to transform the University of the Air envisaged by Harold Wilson in the 1960s to a University of the Cloud, a world-leading institution which is digital by design and has a unique ability to teach and support our students in a way that is responsive both to their needs and those of the economy,” according to Horrocks. As a result the (now leaderless) executive team is working on a ‘transformational model’ for the university, which is still a work in progress.

This is the battlefield into which I parachuted this week.

The eSTEeM conference

The Open University has offered science and technology programs since its inauguration in 1971. It initially used a combination of print, home experiment kits mailed to students’ homes, and one week residential schools in the summer. The residential schools have long since gone (too expensive) although in general students loved them and at least in the early days the residential schools provided such a morale boost for students that many who would have dropped out then went on to continue successfully.

For the last seven years, the STEM Faculty/academic department at the OU has been holding an annual conference to demonstrate the scholarship of its faculty and staff. I was the opening speaker for this year’s conference, on the topic: ‘Digital learning in an era of change: challenges and opportunities for STEM teaching and the OU.’

However, as well as the very interesting STEM components of the conference, on which I will write two separate posts, there was an almost full day, well-organised workshop called ‘Digital by Design’, which focused on what the future as a whole should be at the OU. The workshop enabled a quick and close, if incomplete, ‘parachute’ view of some of the challenges the OU is facing and how academic and regional staff are responding. In this post I will focus on these general, internal challenges that the OU still has to resolve that emerged from this and other discussions in which I participated.

Online but not digital

It is clear that many of the teaching staff have not really ‘got it’ with regard to digital learning. In many cases, print still remains the core teaching technology, and where online is heavily used, it is often just a print model moved online, with a heavy emphasis on content transmission. Many in the OU are still arguing for a ‘blended’ learning model, which in this case refers to a mix of print and online, with print having at least an equal contribution.

In particular, the OU is really weak in its exploitation of the networking and student collaboration that the web offers and in its integration of social media within the design of courses. In this it is not unlike many conventional universities, but nevertheless this realisation came as a real shock to me. This was the original open, distance university, not a conventional one.

Why I am so shocked is that one of the many reasons I emigrated to Canada in 1989 was that I got frustrated at the inability of myself and others at the OU such as Robin Mason and Tony Kaye to get the OU to take online learning seriously. We had contributed to a course, DT200, in 1988 that had an online discussion forum component that had merely been bolted on to the standard 36 week print and broadcast design. The next logical step would have been to have pioneered a fully online course, but neither the university management nor the faculty were interested.

It is important to understand that the OU has a relatively small core of permanent faculty based at its headquarters in Milton Keynes who are primarily engaged in the design of courses, in particular the choice and structuring of content, and a legion of regional staff tutors who provide most of the student learning support. There is a long-established Institute of Educational Technology, where the staff have full academic status, and conduct research as well as advise the OU’s course teams on best practices in the design of distance education.

Here I am 30 years later, and there are still arguments going on about the wisdom of going fully online. This despite the fact that Gilly Salmon, who wrote a standard text on teaching online (2011), worked at the OU for several years, and despite the fact that the OU has an Institute of Educational Technology that has excellent design models developed for online learning that it struggles to get faculty to adopt. This is so reminiscent of Athabasca University and its failure to exploit the expertise of Terry Anderson and its other distance education specialists

The fixation on print as the ‘core’ medium/technology

I participated in several discussions where I challenged the focus on printed material as the core teaching technology. First though I would like to set out some of the arguments OU staff put forward in support of print.

Arguments for print

These were made mainly by OU staff to me.

  1. The OU made its reputation in its early days in the 1970s by the very high quality of its printed materials. As well as being beautifully produced and illustrated (full colour), they were and still are extremely well structured. This was recognised immediately by many faculty in more traditional universities, and the quality of its printed materials is still much appreciated by the students. If it was effective then, it must be effective now.
  2. Access: there are still students in Britain who do not have access to the Internet or cannot afford a computer.
  3. Most OU students are working and many spend all day at work looking at screens; the OU printed material provides an essential break from being on-screen all day.
  4. Students prefer to read printed material; it’s easier for study purposes and revision than searching online.
  5. If the textual material was delivered online rather than printed, the OU would be transferring the cost of print to the students, as they would want to print out the textual material.

Arguments against print

These were made mainly by me to OU staff.

  1. Online learning provides students with the opportunity of ‘any time, any place’ discussion and interaction with each other and teaching staff.
  2. Student activities and interaction with online text is more integrated and immediate than with printed text. In particular immediate feedback can be provided through online tests or automated feedback, etc.
  3. Students are not limited by the boundaries of the printed course material once they go online. Everything on the Internet is potential study material. In particular students can access open educational resources from many different sources.
  4. In order to develop the skills students need in the 21st century, we need to focus more on skills development than on the transmission of content. Online learning can focus better on the development of these soft skills, such as communication and knowledge management.
  5. Access has always been a limitation for any technology. For instance students with visual impairment or dyslexia have difficulties with print. When the OU first started, many students did not have access to the broadcasts. Most students in Britain now have access to the Internet, although in more remote areas there are still bandwidth limitations. The OU’s policy in general has been that when access exceeds 80% of the target audience, alternatives are found for the remaining students. It is wrong to deny the benefits to the vast majority of students because of the needs of a small minority which could be met in other ways.
  6. Students need to learn digitally if they are to earn digitally. Digital literacy is now a core skill required by everyone.
  7. The costs for a print-based system are very high, not just in the actual costs of full colour printing, but in the editing, and above all, the lengthy time it takes faculty and instructors to prepare, check and revise the printed materials (many OU courses take at least two years to design). Savings by going digital could be used to reduce substantially tuition fees.

The need to think digitally when designing online learning

The issue is not whether print has educational value; it does, and there may be specific situations where students may prefer to have hard copy. However, it should not be the default medium. It’s really important when designing online learning to be open to all the media the Internet enables: text, audio, video, computing, augmented reality, simulations, social media, and so on. This requires thinking digitally when designing courses, which is difficult if your first and preferred option is always print.

Of course, this is identical to the challenge that on-campus instructors face about digital learning, but instead of print, their default option is face-to-face teaching.

This is why moving to online learning requires a major cultural change and why it takes so long. However, in the OU’s current existential crisis, it does not have the time for gradual change (that should have started back in 1989). The need for change must be embraced now, ironically, not for financial reasons but for pedagogical reasons: only this way will it better prepare its students for the future. The financial pressures merely make this devastatingly urgent.

Necessary but not sufficient

Forcing change for financial reasons is unlikely to work. Making changes that are not accepted or resisted by staff is more likely to lead to failure or collapse in an organization. Even if by some miracle the (remaining) OU staff manage to pull it off, moving to the University of the Cloud (whatever that means – some kind of heaven for students?) will not meet the needs of the nation that the former OU met.

Lifelong learning is not a luxury but a necessity in a digital age, where the knowledge base expands exponentially and citizens need to continuously learn new content and new skills. Traditional universities do not do lifelong learning well; they are not really designed for it. The OU was, but government policies of starving financial support for part-time learners and reducing the flexibility of study to fit some 1950s view of elite higher education is going to be disastrous for the future British economy. At no time has the OU been more important to Britain. Without a radical change of government policy though its future is indeed dismal, whatever else it does.

Up next

Your intrepid online learning war correspondent will do two more posts from my visit to the OU:

  • the OU’s use of learning analytics for analysing student course evaluations
  • the OU’s use of online labs

Also I will be reporting on a conference on active learning I attended this week at the Royal Military College, Kingston, Ontario. Buy, busy, busy. (Don’t even ask about retirement).

Reference

Salmon, G. (2011). E-moderating: The key to teaching and learning online (3rd ed.). New York: Routledge.

One business case for OER examined

A video on electricity from the OpenLearn platform

Law, P. and Perryman, L.-A. (2017) How OpenLearn supports a business model for OER Distance Education, Vol. 38, No. 1

The journal: ‘Distance Education’

Distance Education is one of the oldest and most established journals in the field. It is the journal of the Open and Distance Learning Association of Australia (ODLAA) and over the years it has published some of the best research in distance education. However, it is not an open access journal, so I am providing my own personal review of one of the articles in this, generally excellent, edition. I should point out though that I am a member of the editorial board so do have an interest in supporting this journal.

Editorial

Som Naidu, the editor, does an excellent job of introducing the articles in the journal under the heading of ‘Openness and flexibility are the norm, but what are the challenges?’ He correctly points out that

While distance education is largely responsible for the articulation and spearheading of openness and flexibility as desirable value principles, these educational goals are fast becoming universally attractive across all sectors and modes of education.

The rapid move to blended and more flexible learning and the slow but increasing use of open educational resources (OER) in campus-based based institutions indeed is challenging the uniqueness of distance education in terms of openness and flexibility. It is easy to argue that distance education is now no more than just another delivery option. Nevertheless, there are still important differences, and Som Naidu draws out some interesting comparisons between the experience of on-campus and distance learning that are still valid.

A business model for OER?

In this latest issue of Distance Education, Patrina Law and Leigh-Anne Perryman have written a very interesting paper about the business case for OER based on three surveys of users (in 2013, 2014, and 2015) of the UK Open University’s OpenLearn project. First some information about OpenLearn:

  • OpenLearn is an open content platform. Initially it used samples of course content from the OU’s undergraduate and postgraduate ‘modules’ (courses) but now hosts specially commissioned audio, video and other interactive materials and short online courses including free certificates and badges;
  • OpenLearn now offers the equivalent of 850 free courses representing 5% of the undergraduate and graduate degree content;
  • 6 million people visit each year with a total of 46 million unique visitors since it was established in 2006; 
  • 13% of users go on to enquire about the OU’s formal degree programs (equivalent of about 1,000 student enrolments per year).

Law and Perryman provide an excellent review of the business cases for OER put forward by others such as the OECD and Creative Commons, then use the survey data from OpenLearn users to test these arguments. Here’s what they found:

  • provision of OER is complementary rather than competitive with the OU’s formal degree programming
  • over half the users are UK-based
  • about 20% reported a disability
  • median age was 36-45
  • about 20% indicated that English was not their first language
  • 70% had some form of post-secondary qualification
  • 16% were part-time or full-time students
  • about two-thirds of the users were ‘tasting’ or ‘testing’ content before making a decision about whether to take a formal program (at either the OU or another institution)
  • almost half (45%) used OpenLearn to find out more about the UK OU (22% had never heard of it before and altogether over half knew nothing or little previously about the OU)
  • the average cost of conversion to OER was between £1500-£2000 per course
  • 13% of OpenLearn users clicked through to make a formal enquiry resulting in about 1,000 new student registrations.

Comment

Very importantly, Law and Perryman link the growing use of OpenLearn to the sudden increase in tuition fees in the UK (£9,000 a year in general, and £5,000 per year for an OU full time degree). Students are not willing to risk this cost without being sure they stand a chance of success and have an interest in the subject. OpenLearn allows them to test this.

This is an important point. The UK government policy of very high tuition fees does appear to be negatively impacting access for many potential students, or at least making them think very carefully before committing to such a large investment. The OU in particular has lost student enrolments as its fees have gone up. There is a danger in my mind that OER can be politically used as a diversion from ‘true’ open education for credit that is available to everyone, irrespective of their means. The best form of open education remains a well-funded state system.

This leads to my one serious criticism of the article. Apart from the cost of conversion, no proper analysis of the true cost of OpenLearn is given so the title is misleading. It does not describe a business model, with full input costs and output benefits stated in monetary terms, but a business case which provides uncosted but positive arguments based on other than cost factors. 

This is a really important distinction because the business model depends heavily on adequate funding for the formal, degree programs which provide the base for the OpenLearn materials. Without that funding, and other costs, OpenLearn will quickly become unsustainable. It is not a parasite in the negative sense of the word but it can’t exist without the funding for the core function of the OU. Without a sense of the full cost of OpenLearn it remains difficult to judge whether the obvious benefits are worth the drain on the OU’s other resources, as the money has to come from somewhere.

Otherwise this is a very good article that should read carefully by anyone concerned with policy regarding the use of OER.

Book review: A History of the Open University

The image is of Robin Wilson, an OU math lecturer (and incidentally the son of Prime Minister Harold Wilson, who helped create the OU). The floral shirt was de rigueur in the early 1970s (I had one myself).

The image is of Robin Wilson, an OU math lecturer (and incidentally the son of Prime Minister Harold Wilson, who helped create the OU). The floral shirt was de rigueur in the early 1970s (I had a few myself).

Weinbren, D. (2105) The Open University: A History Manchester: Manchester University Press/The Open University, 274 pp + notes, £18.99, C$31.61, US$22.30 (paperback edition)

Why you should read this book

From the book cover:

This analysis of the Open University’s precedents, personalities, politics and pedagogies contextualises learners’ experiences and illuminates the change in the values of our society, our ideas about learning and our use of a variety of media.

Despite the florid writing in the publisher’s blurb, this is an accurate summary of the importance of this book, which should be read by anyone interested in open learning, distance learning, equality of access to higher education, changing pedagogies, the role of media in teaching and learning, the politics of creating radically new institutions of higher education, how higher education has changed in terms of value and purpose over the last 45 years, and, most important of all, how open learning can truly transform the lives of individuals.

What the book covers

The book is in four parts, which I will briefly summarise.

Part I: Creating a university of the air

This part covers the origins of the university within the socio-political context of Britain in the 1960s and early 1970s. It is a fascinating story in itself, of how a Labour Prime Minister, Harold Wilson, and Jennie Lee, his Under-Secretary of State in the Department of Education and Science, drove through their vision of a technocratic university for the masses, how the original vision was modified from a University of the Air to a multi-media university, and how the university survived a change of government which brought Margaret Thatcher and the Conservatives to power in the year that the OU opened. This is mandatory reading for policy wonks interested in how to bring about radical change in higher education.

Chapter 1 provides a broad overview of the challenges, developments and successes of the university over its 45 years existence. For those without the time to read the whole book, this chapter provides a useful encapsulation of the main points covered later.

Chapter 2 provides a more detailed account of the creation of the university, covering the political, socio-economic, pedagogical, and media components. It should be noted that many commentators believe that the Open University was ‘the most original innovation in 20th century British higher education’ and a ‘national treasure.’ This chapter helps to explain why.

In this part of the book, Weinbren captures well the social and political conditions, and above all the idealism and philosophies, that underpinned the creation and establishment of the Open University.

Part II: The first two decades

It is one thing to create a new institution; it is quite another to make it work. Indeed, the author notes that other attempts at innovation in higher education, such as the UK Open Polytechnic and the OU e-University, failed dismally after being created.

This part looks in detail at the governance and administrative structures, the role of academics, tutors and counsellors, the pedagogical models, the use of media, and the regional structure.

Weinbren points out that the intention from the start was to develop a degree-granting university with the highest possible academic standards:

Jennie Lee was adamant that the OU should be comparable to other universities in terms of its academic standards, rather than merely representing an educational second chance for the marginalised.

One reason for the OU’s relatively quick acceptance by the rest of the UK higher education sector was the high quality of the course materials which were used extensively by professors (and students) in the other universities. Another reason was the widespread engagement of academics from other universities as tutors or external examiners, who were often initially surprised by the quality of work produced by OU students.

Weinbren addresses particularly well the challenges the OU faced in terms of scale and the need for learner support for students working alone all over the country. (The OU started on day one in 1971 with 25,000 students and has grown since to 150,000 undergraduates and 30,000 postgraduates in 2014, with almost as many taking non-credit courses or modules.)

Weinbren, like the OU itself, struggles with integrating the competing pedagogical philosophies of behaviourist approaches to the design and development of high quality, mass-produced, course materials, and  learner-centred approaches based on face-to-face tutorials and summer schools. Overall, though, he emphasises that the fundamental pedagogical approach of the OU is focused on students developing personal meaning through interaction not only with course materials but also with faculty/tutors and other students, both face-to-face and later online. He describes with clarity how the often changing and complex learner support systems worked.

This chapter also explains why the Conservatives under Mrs. Thatcher continued to support the OU, despite concerns from some Ministers and the national press about ‘Marxist bias’ in its publicly available materials (especially the broadcasts). Thatcher had little time for traditional universities and saw in the OU a chance for reforming and galvanising the higher education system, especially with regard to improved cost-effectiveness.

This chapter also discusses the rapid development and changes in the use of media at the OU, with broadcasting over time playing a less direct role in teaching, greater use of recorded media such as cassettes, and the development of home experiment kits for science and technology courses.

Part III: The OU since the 1990s

This part is much influenced by the many changes since 1990 in the overall higher education system in Britain, and the consequent attempts by the OU at adaptation and accommodation to such changes. The key change has been the shift from direct government funding for teaching and learning to funding largely through student fees supported by loans (the OU now relies almost entirely on student fees), and the attempt by governments (both Labour and Conservative) to introduce more competition between universities and a more ‘market-oriented’ approach. This has resulted in the OU being treated as just any other university by government, rather than the special and separate treatment it received in earlier years.

Also over the past 45 years, the whole HE sector in the UK has expanded rapidly, making access at least theoretically more open to a much higher proportion of the population. Another important development has been the increased use of online learning by conventional universities. Together these have eroded some of the unique differences and advantages of the OU over the rest of the system.

The OU has responded to these changes in a number of ways, including:

  • expanding its international reach, especially but not exclusively in the rest of Europe;
  • the development of continuing education courses and modules;
  • more diversification regionally to respond to national political devolution;
  • contracts with non-commercial agencies, such as the National Health Service, as well as commercial organisations
  • leading the charge to quality assurance processes;
  • moving increasingly to online learning, and in the process, reducing dramatically the high-cost summer schools and face-to-face tutorial support; and integrating the role of counsellors with that of tutors;
  • increased use of learner-centred and project-based learning;
  • creation of open educational resources, such as FutureLearn and BBC/OU programs aimed at the general public.

Although Weinbren does a good job of covering the increasingly diverse and wide-ranging activities of the OU in the years from 1990 to the present, the OU’s unique role and place in the UK HE system becomes inevitably more fuzzy and its future direction less clear. However, the same criticism could apply to the whole of the UK HE sector, which seems to be increasingly forced back to a highly selective and tiered system, by government policies based on a more commercialised and employer-focused view of higher education. The OU’s place in such a system is by no means clear.

Part IV: Half a century of learning

This is a truly wonderful chapter about the student experience at the OU and lets students speak in their own words. This chapter helps explain why the OU is such an iconic component of British culture, and why it is so loved by students and staff alike (it consistently comes out top in student satisfaction in annual surveys of British universities). More importantly, this chapter clearly demonstrates how the OU has changed millions of students lives for the better.

Weinbren looks at several aspects of the student experience. While the OU has a very broad mix of incomes and occupations, it has opened up higher education particularly to working class families, students with disabilities, prisoners, those without high school qualifications, and above all to women. In this sense it is a truly open university, offering not just opportunities but also qualifications and realistic chances of success for everyone.

Weinbren illustrates how important the OU has been to women, particularly in the early year of the OUs, in terms of personal development and increased self-esteem. The importance of summer schools for engaging students and making them feel part of a university community is particularly well described. I also read with great interest how the OU enabled both Republican and Loyalist prisoners in Northern Ireland ‘to develop political confidence and an understanding of methods other than violence’, some going on to become politicians following the Good Friday Agreement of 1998.

Above all, this chapter provides an excellent antidote to the current market-oriented and functional philosophy of higher education now so prevalent in Britain and elsewhere. If you can read only one chapter, this is the one.

Personal reflections on the book

The OU was a very important part of my life for 20 years. I was one of the first staff appointed in 1969, and I ended up doing at one time or another research and evaluation into the educational effectiveness of different media (including the BBC television and radio programs), designing and writing course units, marking student assignments, directing summer schools, and attending endless meetings about policies, directions and the use of media. I left the OU in 1989, partly because I was frustrated that it wasn’t changing fast enough, particularly with regard to the use of online learning. It’s hard for me then to be objective in reviewing Weinbren’s book and even more so in assessing the contribution of the OU to higher education.

Nevertheless, the book captures wonderfully my lived experience of the OU, especially the student response. Weibren has blended together an impeccable range of resources, anecdotes, events and above all personal contributions from academics, staff and students at the OU into a well-written, captivating chronicle that reflects the spirit as well as the history of the OU.

There are criticisms, of course. At times, it becomes a hagiography of an institution (if that’s possible). Weinbren does describe the many criticisms of the OU, but always provides a contradictory positive contribution to offset each criticism. In particular, he could have been harsher about the OU’s increased bureaucracy and sclerosis as it has become older. True, there have been many innovations, for instance, in the use of technology, but changing its cumbersome and now outdated course development system has proved to be extremely difficult. Although it was one of the first institutions to adopt online learning, it has been a real struggle to make it a central rather than a peripheral part of the teaching system.

More importantly, Weinbren does not look into the future, yet there are surely lessons for the future from his book. The OU is facing an almost existential crisis, with many competitors, a very difficult financial situation, and massive changes and innovation going on elsewhere in the UK higher education system. What is the role of the OU in the 21st century? In what ways can it continue to provide a unique and valuable contribution? What teaching model will best meet the needs of its students in the 21st century? This is probably another book altogether, but Weinbren is particularly well placed to ask and address these issues. As I say to Ph.D. students, the conclusion is your chance to let rip and say what you really think now you have established your credentials. It’s a pity that Weinbren did not take this opportunity, but he has probably other means to let his views be known.

These though are minor caveats. Weinbren has undertaken an extremely challenging task and met the challenge superbly. I hope you will enjoy the book as much as I have. More importantly, there are very important lessons to be drawn from this book about the nature of university education, equity, and government policy toward higher education.

Privacy and the use of learning analytics

Image: from Michael Radford's movie, 1984 - Big Brother is watching you!

Image: from Michael Radford’s movie, 1984 – Big Brother is watching you!

Warrell, H. (2105) Students under surveillance Financial Times, July 24

Applications of learning analytics

This is a thoughtful article in the Financial Times about the pros and cons of using learning analytics, drawing on applications from the U.K. Open University, Dartmouth College in the USA, student monitoring service Skyfactor, and CourseSmart, a Silicon Valley start-up that gives universities a window into exactly how e-textbooks are being read.

The UK Open University is using learning analytics to identify students at risk as early as a week into a course.

An algorithm monitoring how much the new recruits have read of their online textbooks, and how keenly they have engaged with web learning forums, will cross-reference this information against data on each person’s socio-economic background. It will identify those likely to founder and pinpoint when they will start struggling. Throughout the course, the university will know how hard students are working by continuing to scrutinise their online reading habits and test scores.

The article also discusses Dartmouth College’s mobile phone app which:

tracks how long students spend working, socialising, exercising and sleeping. The information is used to understand how behaviour affects grades, and to tailor feedback on how students can improve their results.

The article also tries to get a handle on student attitudes to this form of monitoring or surveillance. Not surprisingly, students appear to be somewhat ambiguous about learning analytics and differ in their acceptance of being monitored.

Rationalisations

What was particularly interesting is the range of justifications given in this article for monitoring student behaviour through data analysis:

  • the most obvious is to identify students at risk, so that appropriate interventions can be made. However, there weren’t any examples given in the article of appropriate interventions, highlighting the fact that it is one thing to identify a problem and quite another to know what to do about it. For instance we know that from previous research that students from particular socio-economic backgrounds or students from particular ethnic backgrounds are potentially more at risk than others. What does this mean though in terms of teaching and learning? If you know this is a challenge before students start studying, why wait for learning analytics to identify it as a problem?
  • the next argument is the need to ensure that the high investment each student (or their parents) makes in higher education is not wasted by a failure to complete a program. Because of the high cost, fear of failure is increasing student stress. At Dartmouth, a third of the undergraduate student body saw mental health counsellors last year. However, the solution to that may not be better learning analytics, but finding ways to finance students that don’t lead to such stress in the first place;
  • another rationale is to reduce the financial risk to an institution. The Chief Technology Officer at Skyfactor argues that with revenues from tuition fees of around $25,000+ per student per annum in the USA, avoiding student drop-out is a financial necessity for many U.S. institutions. However, surely there is a moral necessity as well in ensuring that your students don’t fail.

Making sense of learning analytics

The Open University has always collected data on students since it started. In fact, McIntosh, Calder and Smith (1976) found that statistically, the best predictor of success was whether a student returned a questionnaire in the first week of a course, as this indicated their commitment. It still didn’t tell you what to do about the students who didn’t return the questionnaire. (In fact, the OU’s solution at the time was not to count anyone as an enrolment until they had completed an assignment two weeks into the course – advice that MOOC proponents might pay attention to).

As with so many technology developments, the issue is not so much the technology but how the technology is used, and for what purposes. Conscientious instructors have always tried to track or monitor the progress of individual students and learning analytics merely provides a more quantitative and measurable way of tracking progress. The issue though is whether the data you can track and measure can offer solutions when students do run into trouble.

My fear is that learning analytics will replace the qualitative assessment that an instructor gets from, for instance, participating in a live student discussion, monitoring an online discussion forum, or marking assignments. This is more likely to identify the actual conceptual or learning problems that students are having and is more likely to provide clues to the instructor about what needs to be done to address the learning issues. Indeed in a discussion the instructor may be able to deal with it on the spot and not wait for the data analysis. Whether a student chooses to study late at night, for instance, or only reads part of a textbook, might provide a relatively weak correlation with poorer student performance, but recommending students not to stay up late or to read all the textbook may not be the appropriate response for any individual student, and more importantly may well fail to identify key problems with the teaching or learning.

Who gets to use the data?

Which brings me to my last point. Ruth Tudor, president of the Open University’s Students’ Association, reported that:

when the data analytics programme was first mooted, participants were “naturally” anxious about the university selling the information it collected to a third party.

The OU has given strong assurances that it will not do this, but there is growing concern that as higher education institutions come to rely more on direct funding and less government support, they will be tempted to raise revenues by selling data to third parties such as advertisers. As Andrew Keen has argued, this is a particular concern about MOOCs, which rely on other means than direct fees for financial support.

Thus it is incumbent on institutions using learning analytics to have very strong and well enforced policies about student privacy and use of student data. The problem then though is that can easily lead to instructors being denied access to the very data which is of most value in identifying student learning difficulties and possible solutions. Finding the right balance, or applying common sense, is not going to be easy in this area.

Reference

McIntosh, N., Calder, J. and Swift, B. (1976) A Degree of Difference New York: Praeger