July 20, 2017

A better ranking system for university teaching?

Who is top dog among UK universities?
Image: © Australian Dog Lover, 2017 http://www.australiandoglover.com/2017/04/dog-olympics-2017-newcastle-april-23.html

Redden, E. (2017) Britain Tries to Evaluate Teaching Quality Inside Higher Ed, June 22

This excellent article describes in detail a new three-tiered rating system of teaching quality at universities introduced by the U.K. government, as well as a thoughtful discussion. As I have a son and daughter-in-law teaching in a U.K. university and grandchildren either as students or potential students, I have more than an academic interest in this topic.

How are the rankings done?

Under the government’s Teaching Excellence Framework (TEF), universities in England and Wales will get one of three ‘awards’: gold, silver and bronze (apparently there are no other categories, such as tin, brass, iron or dross for those whose teaching really sucks). A total of 295 institutions opted to participate in the ratings.

Universities are compared on six quantitative metrics that cover:

  • retention rates
  • student satisfaction with teaching, assessment and academic support (from the National Student Survey)
  • rates of employment/post-graduate education six months after graduation.

However, awards are relative rather than absolute since they are matched against ‘benchmarks calculated to account for the demographic profile of their students and the mix of programs offered.’ 

This process generates a “hypothesis” of gold, silver or bronze, which a panel of assessors then tests against additional evidence submitted for consideration by the university (higher education institutions can make up to a 15-page submission to TEF assessors). Ultimately the decision of gold, silver or bronze is a human judgment, not the pure product of a mathematical formula.

What are the results?

Not what you might think. Although Oxford and Cambridge universities were awarded gold, so were some less prestigious universities such as the University of Loughborough, while some more prestigious universities received a bronze. So at least it provides an alternative ranking system to those that focus mainly on research and peer reputation.

What is the purpose of the rankings?

This is less clear. Ostensibly (i.e., according to the government) it is initially aimed at giving potential students a better way of knowing how universities stand with regard to teaching. However, knowing the Conservative government in the UK, it is much more likely to be used to link tuition fees to institutional performance, as part of the government’s free market approach to higher education. (The U.K. government allowed universities to set their own fees, on the assumption that the less prestigious universities would offer lower tuition fees, but guess what – they almost all opted for the highest level possible, and still were able to fill seats).

What are the pros and cons of this ranking?

For a more detailed discussion, see the article itself but here is my take on it.

Pros

First this is a more thoughtful approach to ranking than the other systems. It focuses on teaching (which will be many potential students’ initial interest in a university) and provides a useful counter-balance to the emphasis on research in other rankings.

Second it has a more sophisticated approach than just counting up scores on different criteria. It has an element of human judgement and an opportunity for universities to make their case about why they should be ranked highly. In other words it tries to tie institutional goals to teaching performance and tries to take into account the very large differences between universities in the U.K. in terms of student socio-economic background and curricula.

Third, it does provide a simple, understandable ‘award’ system of categorizing universities on their quality of teaching that students and their parents can at least understand.

Fourth, and most important of all, it sends a clear message to institutions that teaching matters. This may seem obvious, but for many universities – and especially faculty – the only thing that really matters is research. Whether though this form of ranking will be sufficient to get institutions to pay more than lip service to teaching remains to be seen.

Cons

However, there are a number of cons. First the national student union is against it, partly because it is heavily weighted by student satisfaction ratings based on the National Student Survey, which thousands of students have been boycotting (I’m not sure why). One would have thought that students in particular would value some accountability regarding the quality of teaching. But then, the NUS has bigger issues with the government, such as the appallingly high tuition fees (C$16,000 a year- the opposition party in parliament, Labour, has promised free tuition).

More importantly, there are the general arguments about university rankings that still apply to this one. They measure institutional performance not individual department or instructor performance, which can vary enormously within the same institution. If you want to study physics it doesn’t help if a university has an overall gold ranking but its physics department is crap or if you get the one instructor who shouldn’t be allowed in the building.

Also the actual quantitative measures are surrogates for actual teaching performance. No-one has observed the teaching to develop the rankings, except the students, and student rankings themselves, while one important measure, can also be highly misleading, based on instructor personality and the extent to which the instructor makes them work to get a good grade.

The real problem here is two-fold: first, the difficulty of assessing quality teaching in the first place: one man’s meat is another man’s poison. There is no general agreement, at least within an academic discipline, as to what counts as quality teaching (for instance, understanding, memory of facts, or skills of analysis – maybe all three are important but can how one teaches to develop these diverse attributes be assessed separately?).

The second problem is the lack of quality data on teaching performance – it just isn’t tracked directly. Since a student may take courses from up to 40 different instructors and from several different disciplines/departments in a bachelor’s program, it is no mean task to assess the collective effectiveness of their quality of teaching. So we are left with surrogates of quality, such as completion rates.

So is it a waste of time – or worse?

No, I don’t think so. People are going to be influenced by rankings, whatever. This particular ranking system may be flawed, but it is a lot better than the other rankings which are so much influenced by tradition and elitism. It could be used in ways that the data do not justify, such as justifying tuition fee increases or decreased government funding to institutions. It is though a first systematic attempt at a national level to assess quality in teaching, and with patience and care could be considerably improved. But most of all, it is an attempt to ensure accountability for the quality of teaching that takes account of the diversity of students and the different mandates of institutions. It may make both university administrations and individual faculty pay more attention to the importance of teaching well, and that is something we should all support.

So I give it a silver – a good try but there is definitely room for improvement. 

Thanks to Clayton Wright for drawing my attention to this.

Next up

I’m going to be travelling for the next three weeks so my opportunity to blog will be limited – but that has been the case for the last six months. My apologies – I promise to do better. However, a four hour layover at Pearson Airport does give me some time for blogging!

The calamitous state of higher education in England and Wales

 

David Cameron (PM) and the Queen, with the Emir of Qatar © The Independent, 2011

I have to be careful when writing about my home country. Having decided to emigrate to Canada just over 20 years ago, a psychological phenomenon known as ‘cognitive dissonance’ kicks in. This is the way we justify our decisions. One example is to dismiss as worthless everything to do with what we rejected in making the decision, and over-praising everything about what we chose. But nevertheless, and even if I betray false consciousness, I despair about what is happening in Britain. Two examples below illustrate this:

Baker, S. (2011) Sharing information on fees ‘might be illegal’, says UUK Times Higher Education, 21 April

Institutions are warned over price-setting as MPs are asked to investigate. Universities in England setting their undergraduate tuition fees for 2012-13 could be breaking competition law simply by sharing information on costs or other “strategically significant matters”, such as support for poorer students.

The warning comes in a briefing by Universities UK on the latest legal advice about the issue, as scrutiny intensifies over the decision by a large number of English institutions to aim for the maximum level of £9,000.

Anything that allows another university to “infer or deduce a pricing or commercial strategy” could be deemed commercially sensitive, according to the briefing, including “financial projections, proposed programme expansion or reduction, and benchmarking”.

Fears over the potential for an Office of Fair Trading investigation into price-setting by universities have been heightened after a union official called on MPs to look into whether institutions were guilty of collusion.

Baker, S. (2011) Deadwood, UK: up to half of courses need cross-subsidy to survive, analysis discovers Times Higher Education, 11 May

At least a third and perhaps up to half of all university courses in the UK are loss-making, and many teaching-led universities have departments with no “meaningful existence” that are being kept afloat by profits from other areas.

These are among the findings of an in-depth analysis of university business models by The Parthenon Group, an international consultancy firm, which says that a large number of institutions have “multiple departments” that are financially unsustainable.

The company’s analysis, which uses data on university costs from the Higher Education Statistics Agency, shows that for a group of about 50 general teaching institutions, a large chunk of their activity is focused on offering five subject areas – business, IT, design, teacher training and nursing. The rest of their resources are spent on several other departments, which are “almost certainly” losing money, Parthenon says.

Mr Robb told THE that the situation had arisen because “budget-based accounting” prevails over commercial thinking; as a result, surpluses from successful courses are invested in “whatever is interesting to academics”.

Well, there we have it. In the UK, higher education is no longer a public good; it is a strictly commercial operation, and universities should operate solely as commercial businesses.

I’m not normally a neo-Marxist, but if you want evidence of the dominance of the neo-liberal agenda, here it is. For those Brits still wanting to emigrate, I’m afraid the lifeboats have already been sold (following a cost-benefit analysis).

[Thanks to Jim Ellis for directing me to this depressing news.]

See also: Has the Anglo-American Academy peaked?

 

Online learning in Britain: what the hell is going on?

White, D. et al. (2010) Study of UK Online Learning Oxford: Department of Continuing Education/HEFCE.

This is a report funded by the Higher Education Funding Council of England and Wales. It is one of the first attempts to measure the extent of online learning in the higher education system of England and Wales.

The study identified over 2,600 HE level online and distance learning courses offered by, or on behalf of, UK HE and FE institutions, between one third and a half of which are delivered online

These included:

  • 1,528 courses offered by 113 HE and FE institutions; of which 510 were identified as being delivered online (including blended learning);
  • 952 courses offered by the Open University
  • 175 courses offered in partnership with commercial partners.

The Open University breakdown is interesting:

  • 257 ‘web supplemented‘ courses (online participation optional);
  • 600 ‘web dependent‘ courses (participation required through interaction with content and/or online communication tools);
  • 95 ‘fully-online‘ courses.

Most fully online courses in this survey were at post-graduate level:

data collected on the 424 courses delivered online without attendance shows that Masters level courses, in particular MAs and MScs, are the most common awards, followed closely by postgraduate diplomas. (This excludes the Open University courses)…..Over half of the UK‟s current offer of ODL courses is at postgraduate level.

They also noted that the majority of ODL courses are in disciplines closely related to professions (i.e. courses for lifelong learners). This resulted in an interesting conclusion:

If the ODL market continues to expand following this emphasis on ‘professional‘ courses, the character of the ODL student body may become increasingly distinct from that of campus-based students. We could be seeing the emergence of a parallel form of HE as a result of this particular mode of delivery.’

The study also found that 175 of the ODL (open and distance learning, but not necessarily online) courses offered were through partnerships between HE institutions and commercial providers (i.e. outsourced to some extent), mainly in the Business area.

The study also included interviews in a smaller sample of institutions to look at the primary issues identified by the institutions around the development and delivery of open and distance education courses. The main conclusion from this section:

The key message to emerge was that institutions felt the substantive challenge was not the pedagogical model they chose to use for ODL, but planning the configuration of the supporting infrastructure, resources and business models required to support the development and delivery of ODL programmes. Addressing these structural issues was seen as a prerequisite for success in expanding provision.

Well, I suspect then they will find the importance of the pedagogical issues when it’s too late.

Methodological issues

This study is beset by methodological issues. Indeed perhaps the most significant finding is this, from the Executive Summary:

‘The study … found that there is no reliable or accurate consolidated source of information about ODL courses offered in the UK that is readily available to students, or other parties, interested in finding ODL programmes, and much of the information on ODL currently remains ‘hidden‘ in labyrinthine institutional websites.’

Another problem here is defining a course. It appears from the report that courses varied considerably in length. Most ‘full’ courses at the Open University are 32 weeks long, thus roughly equal to 2.5 x 3 credit courses in North America, or 7.5 ‘credits’. (Can someone from the UK help me with this? What is the meaning of a ‘course’ these days in conventional universities in the UK? How many Bologna – or UK – credits for example?). However, without knowing how many courses are needed for a Masters, for example, counting courses alone makes it difficult to interpret the findings.

A third problem is the definition of an online course. The authors tried to use something similar to the Sloan definitions, but found that information was often lacking from the institutions to be able to describe accurately what type of online courses they were offering. Their recommendation is very sensible:

For this mode of delivery, it is particularly important to be able to track fully-online courses as these courses are likely to have the widest potential market. We recommend that:


(a) a simple taxonomy of ODL is agreed to act as a framework for data collection;

(b) the HESA data collection process is refined to collect ODL-specific information;

(c) other national methods of data collection, such as the National Student Survey, are reviewed to see if they would be suitable to collect metrics on ODL.

Comments

First congratulations to the HEFCE and the authors for a valiant effort to find out what is actually happening in the UK regarding open and distance learning. This goes much further than anything happening in Canada at the moment.

The report makes clear though that given the vast amount of effort going into using technology for teaching, we need basic data on what types of courses are being offered, in terms of the mix of classroom and technology-based delivery. This data can be collected in a consistent and useful manner only if there are government requirements for institutions to collect and report such data. The main beneficiaries of such data collection will in fact be the institutions themselves, because not only will it enable them to see what is happening within their own institutions over a period of time, it will allow them to ‘benchmark’ against other institutions or jurisdictions. In particular, it is not enough now just to collect figures on ‘distance education’. With blended and hybrid learning as well as fully online learning, we need to know what types of courses are being offered.

Despite the valiant efforts of the researchers, the results left me with more questions than answers. Given the size of England and Wales, these figures for online learning seem very low.

Let’s start with perhaps the most reliable information. In 2010, the Open University is offering only 95 fully online courses. This suggests that the majority of their courses are still print and TV based, or at best a ‘blend’ of print and web. Even allowing for the longer length of Open University courses I find this astonishing, given that it was experimenting with online learning as early as 1988. Some campus-based universities here in Canada have more courses/credits online than the Open University.

The second is that the efforts of the campus-based institutions in the UK to move to online learning seems feeble, to say the least, compared with what’s happening in North American institutions. In particular they do not seem from this study to have appreciated how much technology is going to change the teaching of their mainly on-campus students. For instance, at the University of British Columbia, many of the undergraduate students are taking one or two or three online courses as part of their on-campus studies, because of the flexibility they provide. This is being multiplied in institutions all over North America. The UK survey did not seem to pick up the influence of online learning on campus courses, either. It is true that this is only just beginning to happen in North American, but hybrid courses which change the amount of time spent in classrooms without removing the classroom element altogether will become more and more important, giving more flexibility but more importantly, making better use of the technology and students’ interests. Asking whether courses are ‘web dependent’ does not pick this up.

Of course, and this is a very strong possibility, the study may have grossly underestimated what is happening in UK universities. However, the universities have only themselves to blame for this, burying their online offerings in ‘labyrinthine institutional web sites.’ At least many jurisdictions in North America have one place where students can go to find the range of online courses available from all the institutions within a particular sector, and also within an institution through their Online Learning or Distance Education web pages.

Another report also tries to measure the extent of the e-learning market in the UK:

Patterson, D. et al. (2010) The UK e-learning market 2010 Sheffield UK: e-Learning Centre.

Although the full report will cost you $650 (£500) the executive summary is available free. The report contains not only estimates for the UK market but also for other countries in Europe as well. It is focused mainly on the commercial aspects of e-learning, and forecasts growth of only 6% for 2011, which again seems low compared to North America.

I’d really like to hear from colleagues in the UK on these two reports and whether my comments are totally off the target, because I’m a hell of a long way away!

Thanks to Jim Ellis of the Open University for directing my attention to these two reports; the views though are entirely mine!

UK public universities become privatized while the banks are state-funded

Photo: Prince Charles and Camilla being attacked in their car

You have probably seen the images of Prince Charles and Camilla’s car being attacked by a mob during protests at the British government’s decision to increase undergraduate tuition fees to $14,000 a year (£9,000). Although not condoning the violence, the students are right to be angry.

What the government has done is to cut all university budgets by 40%. Even with tuition fees at $14,000 a year, some analysts fear that universities will not be able to balance their budgets, especially if they do not reach their target numbers, as seems inevitable with fees at this level. For instance a recent report by the Universities and Colleges Union based on research done by two professors at the University of Strathclyde states that:

‘Universities at risk – the impact of cuts in higher education spending on local economies’, published today by UCU, places 49 of England’s 130 higher education institutions at ‘very high’, ‘high’, or ‘high-medium’ risk of serious impact from the proposals, which could leave them vulnerable to merger or, in the worst-case scenario, closure.’

The British Open University, which will suffer the same level of cuts as the other universities, is in a particularly difficult position. Its charter to offer open access to a university education depends on low income students being able to afford its fees.

Part of its strategy will be the same as many other UK universities – look for business opportunities to bring in more revenues to subsidize the rest of the university. This will be a tough task for several reasons. First of all, it’s always had an aggressive marketing policy to generate revenues, so somewhere it has to find completely new markets for its services and products. Second, it will be competing with 142 other universities all in the same boat, with a similar strategy to find ways to increase revenues. Third, as with the other universities, this will detract many of its core faculty away from their basic duties of teaching and research to support marketing initiatives.

Expect to see the brain drain in Britain increase as the best faculty look to move to other countries where they can concentrate on research and teaching. Also expect to see the heavy marketing of online courses in Canada and elsewhere from British institutions desperate for money and trading on their former glory. (In Canada, these used to be called ‘remittance men’: The ‘Remittance Men’ who came to Canada were second sons, which under British tradition of the time meant that these individuals should expect to inherit nothing from their family’s estate.)

Basically, the Conservative-Lib Dem government has withdrawn the state from the funding of undergraduate education. It has turned the universities into businesses dependent on selling services. The irony of this is that at the same time, Britain effectively nationalized the banking system – so the UK now has subsidized banks and private universities. As Gordon Gecko says in Wall Street, it’s the dream of every stock market since the Pharaoh’s – privatize the profits, nationalize the risk. Because the government had to fork out such huge money to rescue the banks, it now has to cut all public services to pay for it – the same as in Ireland.

In the UK’s case, it is particularly invidious that young people, potential students, who had no responsibility whatsoever for the financial crisis, are directly paying for the incompetence and greed of the financial sector. This is the fuel to spark a violent revolution, and I say this with real trepidation as all my family are living and studying or working (so far) in England.

Photo od students rioting, London 2010

Signs of revolution in Britain and what it might mean for us in North America

It’s worth watching carefully what is happening in Britain with university education. As part of its austerity program, the British Conservative-Liberal-Democrat coalition govt is making massive cuts across the whole public sector (mainly because the UK government ran up a huge debt bailing out its banks – its ‘real’ economy is in quite good shape, or it was until the government decided to make the massive cuts). The higher education sector has not been spared.

This e-mail came across my desk today

From: yorksclimatecamp-owner@lists.riseup.net [mailto:yorksclimatecamp-owner@lists.riseup.net] On Behalf Of Andre Pusey

Sent: 20 November 2010 13:24

To: yorksclimatecamp@lists.riseup.net

Subject: [yorksclimatecamp] ‘Reimagine the University’ – 24-26th November

Are you a skint student? An overworked member of University staff? A Leodian with a vision of a more inclusive University system?

The Really Open University invites you to ‘Re-imagine the University’, a three-day event dedicated to exploring and demonstrating an alternative educational system. How can we transform the universities? How can students and lecturers learn differently through more creative, critical and empowering processes? How can higher education institutions benefit their local communities? How do we secure free education for all? Is it even possible to transform the universities?

These are just some of the questions that will be explored in a series of free workshops, lectures, films, plenaries, installations and interventions across the city [presumably York in England]. All events are free and open to everyone.

Background

For a long time the university has been undergoing a process of privatisation. Universities are now run as businesses, with students as consumers and lecturers as creators of products. Knowledge has become a commodity that can be bought and sold. The recent Browne report exacerbates the threat to education with proposals to increase student fees to £9000 a year, while universities face funding cuts of 40%. All this results in students taking on more debt for the same education, lecturers being forced to carry out ‘economies exercises’ and staff working longer and harder hours for less money.

It doesn’t have to be like this. Another University is Possible

The university system is becoming bankrupt and in need of profound change, we need to envisage an alternative, a solution, a way out. As workers and students at different places within the university system, The Really Open University can see a different way forward, we don’t have all the answers, but we have many ideas and are sure there are many more out there.

We would like to explore how universities can become a place where creative and critical thought is fostered, where participants teach what inspires them, learn what they are passionate about, where people share and develop their skills and knowledge in order to create a more equitable and sustainable world, not simply for jobs and profit.

The Really Open University calls on YOU – students, lecturers, university staff and local residents – to come together and demonstrate that another university is possible. It is time to Re-imagine the University. Download the programme here: http://reallyopenuniversity.files.wordpress.com/2010/11/programme-of-events1.pdf

I’m still struggling with this, not the principle that universities need reform and some new visions, but the British government policy of making the user pay full cost, which will almost certainly reduce accessibility and probably lead to large drop in those going on to higher education (which is likely to cost Britain dearly in the future, according to most economists).

The UK policy will almost certainly appeal to the Tea Party in the USA, although they don’t seem to have picked up on it yet (or at least made full cost tuition fees a public issue). Instead, they are pushing the ‘dysfunctionality narrative’, maybe as the artillery barrage before the infantry charge of full cost tuition fees for state universities.

What I’m struggling with is whether there is another route to reform that does not require the privatization of the higher education system aka the UK. Do we need an economic crisis to force change, or is there an alternative way to bring about change in our institutions? Or do we even need major changes rather than just some tidying up?

I’d be interested in your thoughts on this.

See also: Hauptman, A. (2010) Assessing England’s Risky Leap into the Future Chronicle of Higher Education, November 21. Interestingly, Hauptman argues that it would be safer to allow universities to increase enrolments without putting up fees, because of the low marginal costs of taking extra students. I would like to have seen some data supporting this argument, as this is more likely to apply to online than to face-to-face programs. I would not be happy to see online education used as a cheap alternative to classroom teaching. It should stand on its own merits for the students it can help best.