September 21, 2017

A better ranking system for university teaching?

Who is top dog among UK universities?
Image: © Australian Dog Lover, 2017 http://www.australiandoglover.com/2017/04/dog-olympics-2017-newcastle-april-23.html

Redden, E. (2017) Britain Tries to Evaluate Teaching Quality Inside Higher Ed, June 22

This excellent article describes in detail a new three-tiered rating system of teaching quality at universities introduced by the U.K. government, as well as a thoughtful discussion. As I have a son and daughter-in-law teaching in a U.K. university and grandchildren either as students or potential students, I have more than an academic interest in this topic.

How are the rankings done?

Under the government’s Teaching Excellence Framework (TEF), universities in England and Wales will get one of three ‘awards’: gold, silver and bronze (apparently there are no other categories, such as tin, brass, iron or dross for those whose teaching really sucks). A total of 295 institutions opted to participate in the ratings.

Universities are compared on six quantitative metrics that cover:

  • retention rates
  • student satisfaction with teaching, assessment and academic support (from the National Student Survey)
  • rates of employment/post-graduate education six months after graduation.

However, awards are relative rather than absolute since they are matched against ‘benchmarks calculated to account for the demographic profile of their students and the mix of programs offered.’ 

This process generates a “hypothesis” of gold, silver or bronze, which a panel of assessors then tests against additional evidence submitted for consideration by the university (higher education institutions can make up to a 15-page submission to TEF assessors). Ultimately the decision of gold, silver or bronze is a human judgment, not the pure product of a mathematical formula.

What are the results?

Not what you might think. Although Oxford and Cambridge universities were awarded gold, so were some less prestigious universities such as the University of Loughborough, while some more prestigious universities received a bronze. So at least it provides an alternative ranking system to those that focus mainly on research and peer reputation.

What is the purpose of the rankings?

This is less clear. Ostensibly (i.e., according to the government) it is initially aimed at giving potential students a better way of knowing how universities stand with regard to teaching. However, knowing the Conservative government in the UK, it is much more likely to be used to link tuition fees to institutional performance, as part of the government’s free market approach to higher education. (The U.K. government allowed universities to set their own fees, on the assumption that the less prestigious universities would offer lower tuition fees, but guess what – they almost all opted for the highest level possible, and still were able to fill seats).

What are the pros and cons of this ranking?

For a more detailed discussion, see the article itself but here is my take on it.

Pros

First this is a more thoughtful approach to ranking than the other systems. It focuses on teaching (which will be many potential students’ initial interest in a university) and provides a useful counter-balance to the emphasis on research in other rankings.

Second it has a more sophisticated approach than just counting up scores on different criteria. It has an element of human judgement and an opportunity for universities to make their case about why they should be ranked highly. In other words it tries to tie institutional goals to teaching performance and tries to take into account the very large differences between universities in the U.K. in terms of student socio-economic background and curricula.

Third, it does provide a simple, understandable ‘award’ system of categorizing universities on their quality of teaching that students and their parents can at least understand.

Fourth, and most important of all, it sends a clear message to institutions that teaching matters. This may seem obvious, but for many universities – and especially faculty – the only thing that really matters is research. Whether though this form of ranking will be sufficient to get institutions to pay more than lip service to teaching remains to be seen.

Cons

However, there are a number of cons. First the national student union is against it, partly because it is heavily weighted by student satisfaction ratings based on the National Student Survey, which thousands of students have been boycotting (I’m not sure why). One would have thought that students in particular would value some accountability regarding the quality of teaching. But then, the NUS has bigger issues with the government, such as the appallingly high tuition fees (C$16,000 a year- the opposition party in parliament, Labour, has promised free tuition).

More importantly, there are the general arguments about university rankings that still apply to this one. They measure institutional performance not individual department or instructor performance, which can vary enormously within the same institution. If you want to study physics it doesn’t help if a university has an overall gold ranking but its physics department is crap or if you get the one instructor who shouldn’t be allowed in the building.

Also the actual quantitative measures are surrogates for actual teaching performance. No-one has observed the teaching to develop the rankings, except the students, and student rankings themselves, while one important measure, can also be highly misleading, based on instructor personality and the extent to which the instructor makes them work to get a good grade.

The real problem here is two-fold: first, the difficulty of assessing quality teaching in the first place: one man’s meat is another man’s poison. There is no general agreement, at least within an academic discipline, as to what counts as quality teaching (for instance, understanding, memory of facts, or skills of analysis – maybe all three are important but can how one teaches to develop these diverse attributes be assessed separately?).

The second problem is the lack of quality data on teaching performance – it just isn’t tracked directly. Since a student may take courses from up to 40 different instructors and from several different disciplines/departments in a bachelor’s program, it is no mean task to assess the collective effectiveness of their quality of teaching. So we are left with surrogates of quality, such as completion rates.

So is it a waste of time – or worse?

No, I don’t think so. People are going to be influenced by rankings, whatever. This particular ranking system may be flawed, but it is a lot better than the other rankings which are so much influenced by tradition and elitism. It could be used in ways that the data do not justify, such as justifying tuition fee increases or decreased government funding to institutions. It is though a first systematic attempt at a national level to assess quality in teaching, and with patience and care could be considerably improved. But most of all, it is an attempt to ensure accountability for the quality of teaching that takes account of the diversity of students and the different mandates of institutions. It may make both university administrations and individual faculty pay more attention to the importance of teaching well, and that is something we should all support.

So I give it a silver – a good try but there is definitely room for improvement. 

Thanks to Clayton Wright for drawing my attention to this.

Next up

I’m going to be travelling for the next three weeks so my opportunity to blog will be limited – but that has been the case for the last six months. My apologies – I promise to do better. However, a four hour layover at Pearson Airport does give me some time for blogging!

A brighter future for Athabasca University?

Mid-career retraining is seen as one possible focus for Athabasca University’s future

Coates, K. (2017) Independent Third-Party Review of Athabasca University Saskatoon, SK

This report, 45 pages in length plus extensive appendices, was jointly commissioned by the Government of Alberta and the Governors of Athabasca University.

Why the report?

Because Athabasca University, established in 1971 as a fully distance, open university, has been in serious trouble over the last 10 years. In 2015, its Acting President issued a report saying that ‘Athabasca University (AU) will be unable to pay its debt in two years if immediate action is not taken.’ It needed an additional $25 million just to solve its IT problems. Two years earlier, the AU’s senior administrators were savagely grilled by provincial legislators about the financial management of the university, to such an extent that it seemed that the Government of Alberta might well pull the plug on the university.

However, comes a recent provincial election, comes a radical change of government, leading to a new Board and a new President with a five year term. Although these are essential changes for establishing a secure future of the university, in themselves they are not sufficient. The financial situation of the university is temporarily more secure, but the underlying problem of expenses not being matched by revenue remains. It desperately needs more money from a government that is short of revenues since the oil industry tanked. Also its enrolments have started to drop, due to competition from campus-based universities now offering fully online programs. Lastly it still has the same structural problems with an outdated course design and development model and poor student support services, especially on the academic side.

So although the newish government was willing to suspend judgement, it really needed an independent review before shovelling any new money AU’s way – hence this report.

What does the report say?

I will try to summarise briefly the main findings and recommendations, but as always, it is worth reading the full report, which is relatively concise and easy to read:

  • there is substantial student demand in Alberta, across Canada and internationally for AU’s programs, courses and services;
  • the current business model is not financially sustainable and will not support the institution in the coming decades – but ‘it has the potential if significant changes are made to its structure, approach and program mix, to be a viable, sustainable and highly relevant part of the Alberta post-secondary system’;
  • more money is needed to support its operations, especially if it is to remain headquartered in the (small and somewhat remote) Town of Athabasca; the present government funding arrangement is inadequate for the university’s mix of programs and students, especially regarding the support needed for disadvantaged students and those requiring more flexibility in delivery;
  • the emergence of dozens of credible online university alternatives has undermined AU’s competitive advantage – it no longer has a clear and obvious role within the Provincial post-secondary system;
  • AU should re-brand itself as the leading Canadian centre for online learning and 21st century educational technology, but although it has the educational technology professionals needed to provide leadership, it lacks the ICT model and facilities to rise to this opportunity;
  • Open access: AU should expand its activities associated with population groups that are under-represented in the Alberta and Canadian post-secondary system: women in STEM subject, new Canadians, Indigenous Peoples and students with disabilities;
  • diversification of the student body is necessary to achieve economies of scale; in other words it should expand its reach across Canada and internationally and not limit itself just to Alberta;
  • AU should expand its efforts to educate lifelong learners and should expand its career-focused and advanced educational opportunities – particularly mid-career training and training for new work;
  • although there is overwhelming faculty and staff support for AU’s mandate and general approach, there are considerable institutional and financial barriers to effecting a substantial reorientation in AU operations; however, such a re-orientation is critical for its survival.

My comments

Overall, this is an excellent report. Wisely, it does not dwell on the historical reasons why Athabasca University got itself into its current mess but instead focuses on what its future role should be, what it can uniquely contribute to the province, and what is needed to right the ship, including more money.

However, the main challenges, in my view, remain more internal than external. The Board of Governors, senior administration, faculty, staff and students still need to develop together a clear and shared vision for the future of the institution that presents a strong enough value proposition to the government to justify the increased operational and investment funding that is needed. Although the external reviewer does a good job suggesting what some of the elements of such a vision might be, it has to come from the university community itself. This is long overdue and cannot be delayed much longer otherwise the government’s patience will understandably run out. Money itself is not the issue – it is the value proposition that will persuade the government to prioritise funding for AU that still needs to be made by the university itself. In other words it’s a trust issue – if we give you more money, what will you deliver?

The second major challenge, while strongly linked to vision and funding, is the institutional culture. Major changes in course design, educational technology, student support and administration, marketing and PR are urgently needed to bring AU into advanced 21st century practice in online and distance learning. I fear that while there are visionary faculty and staff at AU who understand this, there is still too much resistance from traditionalists and those who see change as undermining academic excellence or threatening their comfort zone. Without these necessary structural and cultural changes though AU will not be able to implement its vision, no matter how persuasive it is. So there is also a competency issue – if we give you more money, can you deliver on your promises?

I think these are still open questions but at least the external review offers a vote of confidence in the university. Now it is up to the university community to turn this opportunity into something more concrete. But it needs to move fast. The window of opportunity is closing fast.

DIY online courses on Facebook

Hughes, M. (2017) Facebook is letting Groups create online learning courses – what could possibly go wrong?  The Next Web, 1 June

It has been a depressing week for me, with terrorist bombings, Trump’s environmental blunder, trade wars and political uncertainty in British Columbia. We are currently without an effective government but that seems not to be a problem for anyone.

However, I did have one good laugh this week that I want to share with you. This came from Matthew Hughes blog (Matthew is at the University of Roehampton, in the UK). He is reporting on news (fake news?) that Facebook is enabling anyone running a Facebook Group to create their own online courses. As he says, tongue in cheek: what could possibly go wrong from an organization with such high standards of credibility?

  • Well, we might begin by looking at the problems that collaborative MOOCs have been facing, especially moderating discussions. Who and how will that moderation be done?
  • Then there’s the likely make-up of the participants  – will this just be more opinion swapping between people with the same world view? 
  • Who will accredit or validate the knowledge in such courses? The learners themselves, probably – and look again at the problems with that in many MOOCs.
  • how will the data that Facebook collects through such Groups be used? You will never know.

This is not to argue that good quality online courses will be impossible through Facebook Group – but once the necessary quality standards are applied, how will it then look different from any other online course delivered on other platforms? What’s the added value?

My point here is that technology alone is never a solution to an educational issue. Do the benefits of the technology (the added value) outweigh the negatives – in this case Big Brother watching you?

One last point while I’m on about Facebook. A Facebook account is a true zombie. You cannot kill it. I ‘signed out’ of Facebook within six months of joining it in 2007, for privacy reasons primarily, and have never used it since, but I still get e-mail messages from them asking me to look at someone I know who has just posted something on Facebook, and I keep getting messages from Facebook to log back in. GO AWAY, FACEBOOK!!!! You are not wanted here.

E-portfolios from Dublin City University to enhance student employability

Dublin City University (2017) DCU launches new online learning portfolio to enhance student employability 24 May

I have been neglecting my blog because I have been really busy with two major projects: a national survey of online and distance education in Canadian post-secondary education; and Contact North’s Pockets of Innovation.

However I came across this news item from Dublin City University, Ireland, which I though was well worth a mention. 

DCU has … launched an online tool [called Reflect] which will allow its students to create a ‘virtual portfolio’ of their academic, professional and personal achievements.  The new platform will provide a lifelong support to DCU students in securing meaningful employment on graduation and remaining employable for the rest of their careers….. 

It is centred around the 6 key graduate attributes (Creative & Enterprising, Solution-Oriented, Effective Communicators, Globally Engaged, Active Leaders, Committed to Continuous Learning) DCU has identified in partnership with employers as being critical to future employability. 

You can get a very brief idea of what the Reflect platform looks like in this video of the project: https://www.youtube.com/watch?v=nMznIkTPUIc&feature=youtu.be

Comment

I will be very interested to see how employers take to this use of e-portfolios. It appears that DCU has gone to some length to consult with employers before launching the platform.

If this is successful, it could really shake up the higher education system of assessment. As an employer I think I would be more impressed with an e-portfolio than a transcript of courses and grades, although of course the two can be used together.

Do you know of any similar use of e-portfolios by post-secondary institutions in North America? And if so, how are they working out?

One business case for OER examined

A video on electricity from the OpenLearn platform

Law, P. and Perryman, L.-A. (2017) How OpenLearn supports a business model for OER Distance Education, Vol. 38, No. 1

The journal: ‘Distance Education’

Distance Education is one of the oldest and most established journals in the field. It is the journal of the Open and Distance Learning Association of Australia (ODLAA) and over the years it has published some of the best research in distance education. However, it is not an open access journal, so I am providing my own personal review of one of the articles in this, generally excellent, edition. I should point out though that I am a member of the editorial board so do have an interest in supporting this journal.

Editorial

Som Naidu, the editor, does an excellent job of introducing the articles in the journal under the heading of ‘Openness and flexibility are the norm, but what are the challenges?’ He correctly points out that

While distance education is largely responsible for the articulation and spearheading of openness and flexibility as desirable value principles, these educational goals are fast becoming universally attractive across all sectors and modes of education.

The rapid move to blended and more flexible learning and the slow but increasing use of open educational resources (OER) in campus-based based institutions indeed is challenging the uniqueness of distance education in terms of openness and flexibility. It is easy to argue that distance education is now no more than just another delivery option. Nevertheless, there are still important differences, and Som Naidu draws out some interesting comparisons between the experience of on-campus and distance learning that are still valid.

A business model for OER?

In this latest issue of Distance Education, Patrina Law and Leigh-Anne Perryman have written a very interesting paper about the business case for OER based on three surveys of users (in 2013, 2014, and 2015) of the UK Open University’s OpenLearn project. First some information about OpenLearn:

  • OpenLearn is an open content platform. Initially it used samples of course content from the OU’s undergraduate and postgraduate ‘modules’ (courses) but now hosts specially commissioned audio, video and other interactive materials and short online courses including free certificates and badges;
  • OpenLearn now offers the equivalent of 850 free courses representing 5% of the undergraduate and graduate degree content;
  • 6 million people visit each year with a total of 46 million unique visitors since it was established in 2006; 
  • 13% of users go on to enquire about the OU’s formal degree programs (equivalent of about 1,000 student enrolments per year).

Law and Perryman provide an excellent review of the business cases for OER put forward by others such as the OECD and Creative Commons, then use the survey data from OpenLearn users to test these arguments. Here’s what they found:

  • provision of OER is complementary rather than competitive with the OU’s formal degree programming
  • over half the users are UK-based
  • about 20% reported a disability
  • median age was 36-45
  • about 20% indicated that English was not their first language
  • 70% had some form of post-secondary qualification
  • 16% were part-time or full-time students
  • about two-thirds of the users were ‘tasting’ or ‘testing’ content before making a decision about whether to take a formal program (at either the OU or another institution)
  • almost half (45%) used OpenLearn to find out more about the UK OU (22% had never heard of it before and altogether over half knew nothing or little previously about the OU)
  • the average cost of conversion to OER was between £1500-£2000 per course
  • 13% of OpenLearn users clicked through to make a formal enquiry resulting in about 1,000 new student registrations.

Comment

Very importantly, Law and Perryman link the growing use of OpenLearn to the sudden increase in tuition fees in the UK (£9,000 a year in general, and £5,000 per year for an OU full time degree). Students are not willing to risk this cost without being sure they stand a chance of success and have an interest in the subject. OpenLearn allows them to test this.

This is an important point. The UK government policy of very high tuition fees does appear to be negatively impacting access for many potential students, or at least making them think very carefully before committing to such a large investment. The OU in particular has lost student enrolments as its fees have gone up. There is a danger in my mind that OER can be politically used as a diversion from ‘true’ open education for credit that is available to everyone, irrespective of their means. The best form of open education remains a well-funded state system.

This leads to my one serious criticism of the article. Apart from the cost of conversion, no proper analysis of the true cost of OpenLearn is given so the title is misleading. It does not describe a business model, with full input costs and output benefits stated in monetary terms, but a business case which provides uncosted but positive arguments based on other than cost factors. 

This is a really important distinction because the business model depends heavily on adequate funding for the formal, degree programs which provide the base for the OpenLearn materials. Without that funding, and other costs, OpenLearn will quickly become unsustainable. It is not a parasite in the negative sense of the word but it can’t exist without the funding for the core function of the OU. Without a sense of the full cost of OpenLearn it remains difficult to judge whether the obvious benefits are worth the drain on the OU’s other resources, as the money has to come from somewhere.

Otherwise this is a very good article that should read carefully by anyone concerned with policy regarding the use of OER.