September 23, 2017

A better ranking system for university teaching?

Who is top dog among UK universities?
Image: © Australian Dog Lover, 2017 http://www.australiandoglover.com/2017/04/dog-olympics-2017-newcastle-april-23.html

Redden, E. (2017) Britain Tries to Evaluate Teaching Quality Inside Higher Ed, June 22

This excellent article describes in detail a new three-tiered rating system of teaching quality at universities introduced by the U.K. government, as well as a thoughtful discussion. As I have a son and daughter-in-law teaching in a U.K. university and grandchildren either as students or potential students, I have more than an academic interest in this topic.

How are the rankings done?

Under the government’s Teaching Excellence Framework (TEF), universities in England and Wales will get one of three ‘awards’: gold, silver and bronze (apparently there are no other categories, such as tin, brass, iron or dross for those whose teaching really sucks). A total of 295 institutions opted to participate in the ratings.

Universities are compared on six quantitative metrics that cover:

  • retention rates
  • student satisfaction with teaching, assessment and academic support (from the National Student Survey)
  • rates of employment/post-graduate education six months after graduation.

However, awards are relative rather than absolute since they are matched against ‘benchmarks calculated to account for the demographic profile of their students and the mix of programs offered.’ 

This process generates a “hypothesis” of gold, silver or bronze, which a panel of assessors then tests against additional evidence submitted for consideration by the university (higher education institutions can make up to a 15-page submission to TEF assessors). Ultimately the decision of gold, silver or bronze is a human judgment, not the pure product of a mathematical formula.

What are the results?

Not what you might think. Although Oxford and Cambridge universities were awarded gold, so were some less prestigious universities such as the University of Loughborough, while some more prestigious universities received a bronze. So at least it provides an alternative ranking system to those that focus mainly on research and peer reputation.

What is the purpose of the rankings?

This is less clear. Ostensibly (i.e., according to the government) it is initially aimed at giving potential students a better way of knowing how universities stand with regard to teaching. However, knowing the Conservative government in the UK, it is much more likely to be used to link tuition fees to institutional performance, as part of the government’s free market approach to higher education. (The U.K. government allowed universities to set their own fees, on the assumption that the less prestigious universities would offer lower tuition fees, but guess what – they almost all opted for the highest level possible, and still were able to fill seats).

What are the pros and cons of this ranking?

For a more detailed discussion, see the article itself but here is my take on it.

Pros

First this is a more thoughtful approach to ranking than the other systems. It focuses on teaching (which will be many potential students’ initial interest in a university) and provides a useful counter-balance to the emphasis on research in other rankings.

Second it has a more sophisticated approach than just counting up scores on different criteria. It has an element of human judgement and an opportunity for universities to make their case about why they should be ranked highly. In other words it tries to tie institutional goals to teaching performance and tries to take into account the very large differences between universities in the U.K. in terms of student socio-economic background and curricula.

Third, it does provide a simple, understandable ‘award’ system of categorizing universities on their quality of teaching that students and their parents can at least understand.

Fourth, and most important of all, it sends a clear message to institutions that teaching matters. This may seem obvious, but for many universities – and especially faculty – the only thing that really matters is research. Whether though this form of ranking will be sufficient to get institutions to pay more than lip service to teaching remains to be seen.

Cons

However, there are a number of cons. First the national student union is against it, partly because it is heavily weighted by student satisfaction ratings based on the National Student Survey, which thousands of students have been boycotting (I’m not sure why). One would have thought that students in particular would value some accountability regarding the quality of teaching. But then, the NUS has bigger issues with the government, such as the appallingly high tuition fees (C$16,000 a year- the opposition party in parliament, Labour, has promised free tuition).

More importantly, there are the general arguments about university rankings that still apply to this one. They measure institutional performance not individual department or instructor performance, which can vary enormously within the same institution. If you want to study physics it doesn’t help if a university has an overall gold ranking but its physics department is crap or if you get the one instructor who shouldn’t be allowed in the building.

Also the actual quantitative measures are surrogates for actual teaching performance. No-one has observed the teaching to develop the rankings, except the students, and student rankings themselves, while one important measure, can also be highly misleading, based on instructor personality and the extent to which the instructor makes them work to get a good grade.

The real problem here is two-fold: first, the difficulty of assessing quality teaching in the first place: one man’s meat is another man’s poison. There is no general agreement, at least within an academic discipline, as to what counts as quality teaching (for instance, understanding, memory of facts, or skills of analysis – maybe all three are important but can how one teaches to develop these diverse attributes be assessed separately?).

The second problem is the lack of quality data on teaching performance – it just isn’t tracked directly. Since a student may take courses from up to 40 different instructors and from several different disciplines/departments in a bachelor’s program, it is no mean task to assess the collective effectiveness of their quality of teaching. So we are left with surrogates of quality, such as completion rates.

So is it a waste of time – or worse?

No, I don’t think so. People are going to be influenced by rankings, whatever. This particular ranking system may be flawed, but it is a lot better than the other rankings which are so much influenced by tradition and elitism. It could be used in ways that the data do not justify, such as justifying tuition fee increases or decreased government funding to institutions. It is though a first systematic attempt at a national level to assess quality in teaching, and with patience and care could be considerably improved. But most of all, it is an attempt to ensure accountability for the quality of teaching that takes account of the diversity of students and the different mandates of institutions. It may make both university administrations and individual faculty pay more attention to the importance of teaching well, and that is something we should all support.

So I give it a silver – a good try but there is definitely room for improvement. 

Thanks to Clayton Wright for drawing my attention to this.

Next up

I’m going to be travelling for the next three weeks so my opportunity to blog will be limited – but that has been the case for the last six months. My apologies – I promise to do better. However, a four hour layover at Pearson Airport does give me some time for blogging!

U.S. university/college financing ‘stabilizing’

Image: © Moody, 2015

Image: © Moody, 2015

Unauthored (2015) Stability and Modest Growth Expected for U.S. Colleges, Inside Higher Ed, December 3

Lederman, D. (2015) ‘Stabilizing’ Financial Picture, Inside Higher Education, July 8

These two reports are for the record (i.e. to help me find the data when I need it for other articles). Nevertheless, they are still interesting.

These are reports of analyses by Moody on the financial status of universities and colleges in the USA. Lederman’s article is about a report released on the financial status of universities and colleges in 2014, and the more recent article is a projection over the next year to 18 months into 2016.

These reports are important, because for the first time since the great U.S. recession in 2008, there is actually overall growth in revenues, especially from state funding, even though tuition revenue is actually declining slightly overall. Nevertheless, the proportion of funding from the state is still considerably less than in 2004, and the situation is not even, with the less prestigious local/state universities still more likely to be in financial trouble than the larger, more prestigious land-grant and private universities.

‘Stabilization’ does not mean that the pressure to reduce the costs of higher education will ease, especially with regard to tuition fees, but it may mean that we will see less media hype about MOOCs and other technology innovations disrupting higher education. Getting costs under control while revenues stabilize will still remain essential, and the more local, less-selective institutions are particularly vulnerable, which is likely to lead to even less equity in the system: to those that have shall it be given.

Next steps for the European HE system

The University of Siena: founded in 1240, but is it still relevant today?

The University of Siena: founded in 1240, but is it still relevant today?

Klemenčič, M. and Ashwin, P. (2015) What’s next for Europe? Inside Higher Ed, May 26

As my holiday in Italy draws to a close, I thought it would be appropriate to do a short blog on developments in European higher education. I look to my many readers in Europe to comment and correct me as appropriate.

What the article is about

This is an interesting article about future policy for European Higher Education, following the Bologna Process Ministerial Conference on May 14-15 in Yerevan, Armenia. (Sigh! Yes, you are right, Armenia is not yet part of the European Union, but it is a member of the Council of Europe, and, since 2005 has been part of the Bologna Process, which sets out pan-European strategy for higher education.)

This article gives a pretty good overview of what the Bologna Process has achieved to date, and also what it has not achieved, and also gives a good description of where European education ministers want to go in the future, in terms of pan-European policy.

The achievements of the Bologna Process

The Bologna Process is:

a voluntary convergence and coordinated reform of higher education systems across the member countries of the European Union and beyond. The aims have been to promote the mobility of students and staff and to enhance the quality and international competitiveness of European higher education.

The European Higher Education Area (EHEA) now has 48 members, including non-European Union countries such as the Russian Federation, Belarus, Armenia and the Ukraine.

Its successes include:

  • a common three-cycle degree structure across countries;
  • student mobility: students can transfer course credits acquired at one institution to any another institution in the EHEA;
  • European Standards and Guidelines for Quality Assurance and the European Register of Quality Assurance Agencies, ensuring that all countries have compatible internal and external quality assurance procedures.

This of course raises the question: If the Europeans can enable students to transfer seamlessly between 4,000 higher education institutions across Europe and beyond (and they do, in droves), why is it so difficult to do this in Canada, and particularly within Ontario, for God’s sake?

One of the great scandals of Canadian higher education is the refusal of Ontario universities or the Ontario government to put in place any form of automatic transfer of credits. (Yeah, there are seven universities in Ontario that have a paper agreement amongst themselves, but the reality is that it is NOT an automatic process even between these seven institutions).

BC and Alberta have had a mutual transfer system in place for many years, but the only thing more difficult than moving from a university in BC (or from anywhere else in Canada) to a university in Ontario is taking a bottle of BC wine with you to Ontario (yes, that is actually illegal in Canada). Talk about provincialism.

The challenges of the Bologna Process

Like anything to do with the European Union, excessive bureaucracy is a major challenge. In particular, to quote from the article:

much of the energy of the Bologna Follow-Up Group, the governing body of the process, has been channeled into detailed questions about decision structures and processes. The Bologna Process needed a new sense of purpose to bring the governments together and re-energize international cooperation within the EHEA. And this indeed happened [at the Yerevan conference].

What’s driven this new sense of purpose is youth unemployment:

The unemployment rate for people 29 and younger in the European Union is 19 percent, the highest in at least 10 years. In Spain, the figure was 53 percent in November 2014; it was 49 percent, in Greece, followed closely by Croatia and Italy. Higher education is seen as one key pillar in Europe’s vision to fight unemployment among young people, preventing them from becoming a “lost generation” and source of social upheaval. The communiqué emphasizes the need to ensure that graduates possess competencies that will make them employable.

The article lists several ways this is to be done, such as:

  • a better dialogue between higher education institutions and employers,
  • a good balance between theoretical and practical components in curricula, and
  • continued support for international mobility for study and work placement.

The authors though acknowledge that:

higher education alone, of course, cannot solve the problem that is so clearly linked to economic growth and also labor regulations.

They might also have mentioned the failed economic policy of austerity, which is a major cause of youth unemployment in Europe.

A second objective is to make European higher education more inclusive. A particular concern is the low participation rate of immigrants from the Middle East and Africa in European higher education, and the possible radicalization of immigrant youth:

three types of mobility are accentuated in the communiqué: for students and staff from conflict areas, students from disadvantaged backgrounds, and mobility of teacher education students.

The third objective is to improve the quality of teaching and learning in universities and colleges. The authors note that it is surprising that it has taken so long for this to emerge as a priority for this first time at the Armenia conference:

..the quality of teaching and learning is far from satisfactory and varies significantly across European systems and institutions…the majority of countries do not have a strategy for the advancement of teaching and learning or specific structures to support it. At best, higher education institutions are developing their own units for supporting excellence in teaching and learning or funding teaching development programs. At worse, higher education teachers are left to their own devices to improve their teaching (or not) when alerted by the outcomes of student satisfaction surveys.

In Yerevan, the ministers have committed to support higher education institutions in pedagogical innovation, exploring the use of digital technologies for learning and teaching, and in better linking learning and teaching with research, innovation and entrepreneurship. You have to wonder though why it took almost 20 years to get these items on the agenda.

What next?

The authors of the article are surprisingly optimistic that these new policies will be successfully implemented by the governments of member states. However, by 2018, the set target gate for implementation, both Greece and Britain may well have left the European Union, and I will be surprised if countries such as Italy, the Ukraine and Bulgaria will have made much progress towards these objectives, because of structural and economic difficulties.

Nevertheless, on balance, despite the stifling bureaucracy of the European Union, and the political and economic challenges faced by many European countries, the Bologna Process has enabled many European universities to improve their standards and to modernise, and is likely to continue to do so into the future.

One of the University of Siena's student computer labs

One of the University of Siena’s student computer labs