August 22, 2017

A better ranking system for university teaching?

Who is top dog among UK universities?
Image: © Australian Dog Lover, 2017 http://www.australiandoglover.com/2017/04/dog-olympics-2017-newcastle-april-23.html

Redden, E. (2017) Britain Tries to Evaluate Teaching Quality Inside Higher Ed, June 22

This excellent article describes in detail a new three-tiered rating system of teaching quality at universities introduced by the U.K. government, as well as a thoughtful discussion. As I have a son and daughter-in-law teaching in a U.K. university and grandchildren either as students or potential students, I have more than an academic interest in this topic.

How are the rankings done?

Under the government’s Teaching Excellence Framework (TEF), universities in England and Wales will get one of three ‘awards’: gold, silver and bronze (apparently there are no other categories, such as tin, brass, iron or dross for those whose teaching really sucks). A total of 295 institutions opted to participate in the ratings.

Universities are compared on six quantitative metrics that cover:

  • retention rates
  • student satisfaction with teaching, assessment and academic support (from the National Student Survey)
  • rates of employment/post-graduate education six months after graduation.

However, awards are relative rather than absolute since they are matched against ‘benchmarks calculated to account for the demographic profile of their students and the mix of programs offered.’ 

This process generates a “hypothesis” of gold, silver or bronze, which a panel of assessors then tests against additional evidence submitted for consideration by the university (higher education institutions can make up to a 15-page submission to TEF assessors). Ultimately the decision of gold, silver or bronze is a human judgment, not the pure product of a mathematical formula.

What are the results?

Not what you might think. Although Oxford and Cambridge universities were awarded gold, so were some less prestigious universities such as the University of Loughborough, while some more prestigious universities received a bronze. So at least it provides an alternative ranking system to those that focus mainly on research and peer reputation.

What is the purpose of the rankings?

This is less clear. Ostensibly (i.e., according to the government) it is initially aimed at giving potential students a better way of knowing how universities stand with regard to teaching. However, knowing the Conservative government in the UK, it is much more likely to be used to link tuition fees to institutional performance, as part of the government’s free market approach to higher education. (The U.K. government allowed universities to set their own fees, on the assumption that the less prestigious universities would offer lower tuition fees, but guess what – they almost all opted for the highest level possible, and still were able to fill seats).

What are the pros and cons of this ranking?

For a more detailed discussion, see the article itself but here is my take on it.

Pros

First this is a more thoughtful approach to ranking than the other systems. It focuses on teaching (which will be many potential students’ initial interest in a university) and provides a useful counter-balance to the emphasis on research in other rankings.

Second it has a more sophisticated approach than just counting up scores on different criteria. It has an element of human judgement and an opportunity for universities to make their case about why they should be ranked highly. In other words it tries to tie institutional goals to teaching performance and tries to take into account the very large differences between universities in the U.K. in terms of student socio-economic background and curricula.

Third, it does provide a simple, understandable ‘award’ system of categorizing universities on their quality of teaching that students and their parents can at least understand.

Fourth, and most important of all, it sends a clear message to institutions that teaching matters. This may seem obvious, but for many universities – and especially faculty – the only thing that really matters is research. Whether though this form of ranking will be sufficient to get institutions to pay more than lip service to teaching remains to be seen.

Cons

However, there are a number of cons. First the national student union is against it, partly because it is heavily weighted by student satisfaction ratings based on the National Student Survey, which thousands of students have been boycotting (I’m not sure why). One would have thought that students in particular would value some accountability regarding the quality of teaching. But then, the NUS has bigger issues with the government, such as the appallingly high tuition fees (C$16,000 a year- the opposition party in parliament, Labour, has promised free tuition).

More importantly, there are the general arguments about university rankings that still apply to this one. They measure institutional performance not individual department or instructor performance, which can vary enormously within the same institution. If you want to study physics it doesn’t help if a university has an overall gold ranking but its physics department is crap or if you get the one instructor who shouldn’t be allowed in the building.

Also the actual quantitative measures are surrogates for actual teaching performance. No-one has observed the teaching to develop the rankings, except the students, and student rankings themselves, while one important measure, can also be highly misleading, based on instructor personality and the extent to which the instructor makes them work to get a good grade.

The real problem here is two-fold: first, the difficulty of assessing quality teaching in the first place: one man’s meat is another man’s poison. There is no general agreement, at least within an academic discipline, as to what counts as quality teaching (for instance, understanding, memory of facts, or skills of analysis – maybe all three are important but can how one teaches to develop these diverse attributes be assessed separately?).

The second problem is the lack of quality data on teaching performance – it just isn’t tracked directly. Since a student may take courses from up to 40 different instructors and from several different disciplines/departments in a bachelor’s program, it is no mean task to assess the collective effectiveness of their quality of teaching. So we are left with surrogates of quality, such as completion rates.

So is it a waste of time – or worse?

No, I don’t think so. People are going to be influenced by rankings, whatever. This particular ranking system may be flawed, but it is a lot better than the other rankings which are so much influenced by tradition and elitism. It could be used in ways that the data do not justify, such as justifying tuition fee increases or decreased government funding to institutions. It is though a first systematic attempt at a national level to assess quality in teaching, and with patience and care could be considerably improved. But most of all, it is an attempt to ensure accountability for the quality of teaching that takes account of the diversity of students and the different mandates of institutions. It may make both university administrations and individual faculty pay more attention to the importance of teaching well, and that is something we should all support.

So I give it a silver – a good try but there is definitely room for improvement. 

Thanks to Clayton Wright for drawing my attention to this.

Next up

I’m going to be travelling for the next three weeks so my opportunity to blog will be limited – but that has been the case for the last six months. My apologies – I promise to do better. However, a four hour layover at Pearson Airport does give me some time for blogging!

A brighter future for Athabasca University?

Mid-career retraining is seen as one possible focus for Athabasca University’s future

Coates, K. (2017) Independent Third-Party Review of Athabasca University Saskatoon, SK

This report, 45 pages in length plus extensive appendices, was jointly commissioned by the Government of Alberta and the Governors of Athabasca University.

Why the report?

Because Athabasca University, established in 1971 as a fully distance, open university, has been in serious trouble over the last 10 years. In 2015, its Acting President issued a report saying that ‘Athabasca University (AU) will be unable to pay its debt in two years if immediate action is not taken.’ It needed an additional $25 million just to solve its IT problems. Two years earlier, the AU’s senior administrators were savagely grilled by provincial legislators about the financial management of the university, to such an extent that it seemed that the Government of Alberta might well pull the plug on the university.

However, comes a recent provincial election, comes a radical change of government, leading to a new Board and a new President with a five year term. Although these are essential changes for establishing a secure future of the university, in themselves they are not sufficient. The financial situation of the university is temporarily more secure, but the underlying problem of expenses not being matched by revenue remains. It desperately needs more money from a government that is short of revenues since the oil industry tanked. Also its enrolments have started to drop, due to competition from campus-based universities now offering fully online programs. Lastly it still has the same structural problems with an outdated course design and development model and poor student support services, especially on the academic side.

So although the newish government was willing to suspend judgement, it really needed an independent review before shovelling any new money AU’s way – hence this report.

What does the report say?

I will try to summarise briefly the main findings and recommendations, but as always, it is worth reading the full report, which is relatively concise and easy to read:

  • there is substantial student demand in Alberta, across Canada and internationally for AU’s programs, courses and services;
  • the current business model is not financially sustainable and will not support the institution in the coming decades – but ‘it has the potential if significant changes are made to its structure, approach and program mix, to be a viable, sustainable and highly relevant part of the Alberta post-secondary system’;
  • more money is needed to support its operations, especially if it is to remain headquartered in the (small and somewhat remote) Town of Athabasca; the present government funding arrangement is inadequate for the university’s mix of programs and students, especially regarding the support needed for disadvantaged students and those requiring more flexibility in delivery;
  • the emergence of dozens of credible online university alternatives has undermined AU’s competitive advantage – it no longer has a clear and obvious role within the Provincial post-secondary system;
  • AU should re-brand itself as the leading Canadian centre for online learning and 21st century educational technology, but although it has the educational technology professionals needed to provide leadership, it lacks the ICT model and facilities to rise to this opportunity;
  • Open access: AU should expand its activities associated with population groups that are under-represented in the Alberta and Canadian post-secondary system: women in STEM subject, new Canadians, Indigenous Peoples and students with disabilities;
  • diversification of the student body is necessary to achieve economies of scale; in other words it should expand its reach across Canada and internationally and not limit itself just to Alberta;
  • AU should expand its efforts to educate lifelong learners and should expand its career-focused and advanced educational opportunities – particularly mid-career training and training for new work;
  • although there is overwhelming faculty and staff support for AU’s mandate and general approach, there are considerable institutional and financial barriers to effecting a substantial reorientation in AU operations; however, such a re-orientation is critical for its survival.

My comments

Overall, this is an excellent report. Wisely, it does not dwell on the historical reasons why Athabasca University got itself into its current mess but instead focuses on what its future role should be, what it can uniquely contribute to the province, and what is needed to right the ship, including more money.

However, the main challenges, in my view, remain more internal than external. The Board of Governors, senior administration, faculty, staff and students still need to develop together a clear and shared vision for the future of the institution that presents a strong enough value proposition to the government to justify the increased operational and investment funding that is needed. Although the external reviewer does a good job suggesting what some of the elements of such a vision might be, it has to come from the university community itself. This is long overdue and cannot be delayed much longer otherwise the government’s patience will understandably run out. Money itself is not the issue – it is the value proposition that will persuade the government to prioritise funding for AU that still needs to be made by the university itself. In other words it’s a trust issue – if we give you more money, what will you deliver?

The second major challenge, while strongly linked to vision and funding, is the institutional culture. Major changes in course design, educational technology, student support and administration, marketing and PR are urgently needed to bring AU into advanced 21st century practice in online and distance learning. I fear that while there are visionary faculty and staff at AU who understand this, there is still too much resistance from traditionalists and those who see change as undermining academic excellence or threatening their comfort zone. Without these necessary structural and cultural changes though AU will not be able to implement its vision, no matter how persuasive it is. So there is also a competency issue – if we give you more money, can you deliver on your promises?

I think these are still open questions but at least the external review offers a vote of confidence in the university. Now it is up to the university community to turn this opportunity into something more concrete. But it needs to move fast. The window of opportunity is closing fast.

Corruption in higher education: a wake-up call

Staff at Pavol Jozef Safarik University, Kosice, Slovakia were accused of taking bribes to admit students to its Medical School

Staff at Pavol Jozef Safarik University, Kosice, Slovakia have been accused of taking bribes to admit students to the Medical School

Daniel, J. (2016) Combatting Corruption and Enhancing Integrity: A Contemporary Challenge for the Quality and Integrity of Higher Education: Advisory Statement for Effective International Practice: Washington DC/Paris: CHEA/UNESCO

Daniel, J. (2016) Lutter contre la corruption et renforcer l’intégrité : un défi contemporain pour la qualité et la crédibilité de l’enseignement supérieur: Déclaration consultative pour des pratiques internationales efficaces Washington DC/Paris: CHEA/UNESCO

Those of us working in online learning are often berated by academic colleagues about the possible lack of integrity in online learning due to issues such as plagiarism, diploma mills, or ‘easy’ qualifications lacking rigorous academic process. Such cases do occur, but having read this document, it seems that the more traditional areas of higher education are prone to far more egregious forms of corruption.

Where do we find corruption?

At the end of this report, there is a list of references chronicling corruption in higher education in Australia, China, the Czech Republic, Egypt, France, Germany, India, Kenya, Nigeria, Russia, Slovakia, South Africa, and the USA. And those are just the ones who have been recently caught.

The report puts it bluntly:

This Advisory Statement is a wake-up call to higher education worldwide – particularly to quality assurance bodies. HEIs [higher education institutions], governments, employers and societies generally, in both developed and developing countries, are far too complacent about the growth of corrupt practices, either assuming that these vices occur somewhere else or turning a deaf ear to rumours of malpractice in their own organizations.

What kinds of corruption?

You name it, it’s in this report. In fact, the report describes 29 different kinds of corrupt practices. Here are just a few examples:

  • giving institutions licenses, granting degree-awarding powers, or accrediting programmes in return for bribes or favours.
  • altering student marks in return for sexual or other favours.

  • administrative pressure on academics to alter marks for institutional convenience.

  • publishing false recruitment advertising.

  • impersonation of candidates and ghost writing of assignments.

  • political pressures on higher education institutions to award degrees to public figures.

  • publication by supervisors of research by graduate students without acknowledgement.

  • higher education institutions publishing misleading news releases or suppressing inconvenient news.

Who is sounding the alarm?

Although the writer of the report is Sir John Daniel, a fellow Research Associate at Contact North, and former Vice-Chancellor, the Open University, Assistant Director-General for Education at UNESCO and President of the Commonwealth of Learning, the report draws on meetings of expert groups from the following organizations:

  • UNESCO’s International Institute for Educational Planning (IIEP)
  • the International Quality Group of the US Council for Higher Education Accreditation (CHEA/CIQG).

What’s causing this?

Corruption is as much about lack of ethical behaviour and rampant self-interest as about policies and practices. The report though points to two key factors that are contributing to corruption:

  • the huge appetite for higher education among the young populations of the developing world puts great pressures on admissions processes;
  • the steadily developing sophistication and borderless nature of information and communications technology (ICT) has expanded the opportunities for fraudsters in all walks of life.

What are the recommended solutions?

There are of course no easy solutions here. The report points out that there are both ‘upstream’ possibilities for corruption at the level of government and accrediting agencies, and downstream, from individuals desperate to get into and succeed within an increasingly competitive higher education system. In the middle are the institutions themselves.

The report separates its recommendations for combatting corruption then into several target areas:

  1. the regulation of higher education systems
  2.  the teaching role of higher education institutions
  3. student admissions and recruitment
  4. student assessment
  5. credentials and qualifications
  6. research theses and publications
  7. through increased public awareness

It is interesting that while the report emphasizes the importance of internal quality assurance processes within HEIs, it also notes that the more ‘mature’ an HE system becomes, the more external quality assurance agencies, such as accreditation boards and government ministries, tend to pass quality assurance responsibilities back to the institutions. The report notes that students themselves have a very important role to play in demanding transparency and whistle-blowing.

A call to action

The report ends with the following:

  • governments, quality assurance agencies and HEIs worldwide must become more aware of the threat that corruption poses to the credibility, effectiveness and quality of higher education at a time when its importance as a driver of global development has never been higher.

  • external quality assurance agencies should do more to review the risks of corruption in their work and HEIs must ensure that their IQA [internal quality assurance] frameworks are also fit for the purpose of combatting corruption.

  • training and supporting staff in identifying and exposing corrupt practices should be stepped up.

  • creating networks of organizations that are fighting corruption and greater North-South collaboration in capacity building for this purpose are highly desirable.

So next time some sanctimonious academic sneers at the academic integrity of online learning, just point them in the direction of this report.

Athabasca University’s troubles grow

Athabasca University: a campus without students - or even resident faculty

Athabasca University: a campus without students – or even resident faculty

Bako, O. (2015) AU taskforce releases sustainability report, The Athabasca Advocate, June 9

I hate to rely on third-party reports, but a thorough web search did not find the actual sustainability report online, so all I can do is provide a summary of the newspaper article. (How comes an open university doesn’t make this most important report ever on its future available online? This is just another indicator of the depth of the problems at AU. It doesn’t understand the present, never mind the future). If anyone can supply me with a copy of the actual report, I would appreciate it. So until I get a copy, please see this as an interim post.

What’s the problem?

However you look at it, though, Canada’s largest open university is in an existential crisis:

Athabasca University (AU) will be unable to pay its debt in two years if immediate action is not taken, according to a sustainability report released June 1….the taskforce concluded AU would face insolvency on its debt in two years….’Our problem is a common one — our expenses are going up faster than our revenues,’ said AU interim president Peter MacKinnon.

So Athabasca University is now in the same position as the Greek government, except it doesn’t have the EU, the IMF, or the Germans to look to for help – just the Alberta government, which itself has been fiscally devastated by the collapse of oil prices.

What’s it doing about it?

From the newspaper report, though, the task force is looking forward, rather than backwards. It has looked at four options, which are not mutually exclusive:

  • increase cost-effectiveness and streamline business, administrative and education practices;
  • refocus AU’s mission to serve only, or primarily, students who physically reside in Alberta;
  • become a federation with another Alberta post-secondary institution and/or
  • join an affiliation with universities both in and out of the province (e.g. the Canadian Virtual University).

(I probably have mixed up the options, but you can see what kind of options are being considered).

Why is it in such a mess?

I’m not sure I’m the right person to answer this question, nor the newspaper, but the interim President seems to offer up some culprits:

  • increased reliance on tuition and decreased government funding to cover operating expenses,
  • caps on tuition that limit AU revenue,
  • the location of AU (its main campus is in the boondocks 200 kilometres north of Edmonton and most of the faculty live elsewhere),
  • demands from employee collective agreements,
  • the province does not see IT infrastructure as a capital expense.

However, none (except possibly the over-generous and/or inflexible collective agreements) of these really gets to the root of the problem. There are three fundamental reasons for AU’s problems, all strongly interconnected:

  • gross financial and political mismanagement by the previous university administration (Board and President);
  • a failure by the previous Alberta government to understand or support a unique institution;
  • a failure to adapt fast enough from a heavily front-end loaded, print-based teaching model to lighter, more flexible online teaching methods.

What will happen to AU?

God knows, and certainly not someone sitting here in Vancouver. However, I have the heavy foreboding of a tragedy rapidly unfolding. Here’s why:

  • doing the same as it has been, but more ‘efficiently’, i.e. more cheaply, won’t cut it. The only way to really reduce costs within the current teaching model is to cut student support and central faculty, which will mean lower completion rates and/or poorer quality courses; it needs to implement a completely different model of teaching and learning, which would reduce operating costs, but would also require major capital investment, and ripping up current collective agreements, neither of which appear to be options;
  • its current model has high fixed costs and low marginal costs; limiting enrolment to Albertans will leave the high fixed costs, but will reduce the additional income from external students, a recipe for disaster. Indeed, AU should go in the opposite direction and go global – but it will need a more up-to-date teaching model to do this;
  • I find it hard to believe that the Albert government doesn’t understand that investment in IT infrastructure is essential for a modern university, and is far cheaper than building new campuses. More likely it is hesitant to throw good money after bad, because AU has previously poorly managed the management of IT.

One tempting solution would be to close Athabasca, and give the two main universities a mandate to absorb and redevelop the programs Athabasca has been offering, using some of the money that is currently being invested in Athabasca. However, there are two problems with this temptation:

  • first, Alberta’s problem, compared to other provinces, is that it has in the past left online and distance learning to Athabasca University. As a result, its other institutions, such as the University of Alberta and the University of Calgary, are way behind many other universities in Canada in moving to online and blended learning. They therefore lack current capacity. On the other hand, it could be argued that by hiring the best from Athabasca, this would be a good way to bring Alberta’s conventional universities up to speed in online learning. My guess though is that neither the University of Alberta nor Calgary would see this as a priority or even necessary. Alberta is not the only province with a degree of hubris.
  • Second, Athabasca is not really an online university, it is an open university, and in that sense it is unique. The British Columbia government’s experience in closing the Open Learning Agency and with it the Open University of BC in 2003 suggests that it will be very difficult to integrate the open components within a conventional university, as the experience at Thompson Rivers University, which now offers open courses through its Open Learning division, indicates.

What would I propose?

Thank goodness, it’s not my responsibility, and I don’t have all the facts and information needed to make an informed decision. But here are some suggestions:

  • this is really the Alberta government’s decision. It surely now has enough evidence to know that the current situation cannot continue and that there is no easy solution short of closing down Athabasca, which then opens up a whole new can of worms
  • what Alberta lacks (and it is not alone in this) is a provincial wide strategy for online and open learning. There is clearly demand, in terms of student numbers, and opportunities, in terms of online, blended learning, OERs and open access. What is the best way to provide this, in terms of the roles of the conventional institutions, and where does Athabasca then fit, if at all? Until it has worked out such a strategy, it should hold off on any interim decisions about Athabasca, other than to keep it on life support.

However, don’t bank on logic or strategic thinking to drive decision-making about Athabasca University. There are jobs in rural areas at stake, as well as the future of many dedicated and skilled faculty and staff at Athabasca University. But this is where Alberta’s real resources lie, in the skills and knowledge buried within Athabasca University, not in a particular institutional arrangement. So the government needs to act with caution but the goal should be making Alberta a leader in online and open education, by drawing on and possibly relocating the expertise within Athabasca, but not necessarily supporting the institution itself whose time may well have passed.

 

UBC develops an institutional strategy for learning technologies

The Amalfi Coast

The Amalfi Coast, Italy

Bates, S. et al. (2015) UBC’s Learning Technology Ecosystem: Developing a Shared Vision, Blueprint & Roadmap Vancouver BC: University of British Columbia

I’ve not been posting much recently as I am taking a three week holiday in Europe (the photo is my view as I write this), but this report from the Provost’s Office at UBC is too significant to ignore.

What is it about?

The report basically sets out a vision and a set of strategies for the future development and management of learning technologies at UBC, a large Tier-1 research university in Canada. Although produced by a small Project Committee, it is the outcome of extensive discussions throughout the university and also externally with other institutions with successful learning technology strategies.

What is in the report?

1. Recognition of learning technology as an eco-system

A learning technology ecosystem represents faculty, staff and students interacting with their learning technology environment, composed of tools and services. There are dependencies in this ecosystem; between technologies, between technologies and services but also between users, technologies and services. The ecosystem is self-organizing, dynamic, constantly changing and evolving. Technologies are birthed, and they also are removed as new ones take their place.

2. Assessment of the current state of learning technologies

UBC uses a very interesting way of assessing the current state of learning technology within an institution, using the following conceptual framework:

UBC's current state assessment process framework

UBC’s current state assessment process framework

This has enabled the team to identify gaps in services, governance, funding and infrastructure.

Another interesting outcome of this process is that the report estimates that UBC is currently spending almost $10 million annually on supporting its LMS, of which 78% is incurred at a Faculty/academic department level, mainly in technical support for the LMS, the rest centrally, including licensing. Thus one technology tool is costing almost as much as the rest of the LT eco-system.

2. Vision and principles for LTs at UBC

UBC's LT vision and principles 2

3. Functions and services

Working group members identified functional gaps in the LT ecosystem, along with their relative importance. Similarly, members of the Working Group identified both phase-specific support required during LT life cycles, as well as support services required across the lifecycle. They identified which of the gaps required the most improvement and also prioritized them according to their relative importance.

4. Support models

UBC uses both central and local/departmental support models and because of the size and complexity of the organization, no major changes were suggested for support models (but see Governance below)

5. Governance

The working group found significant shortcomings in the current governance structure for LTs. In particular there was inadequate academic input into priorities for the selection and use of LT tools and services, and the student voice was not heard. The Working Group proposed a stronger governance model as a result.

6. Other issues

The report goes on to cover a number of other issues, such as a roadmap and success metrics and resource issues such as the need for better learning analytics and increased bandwidth.

Why this report?

Good question, Tony, and here I will have to speculate a little, as I no longer work at UBC. UBC has a long history in both distance education and learning technology development. In the early 1990s it received government funding of over $2 million to explore the use of learning technologies, one outcome of which was WebCT, the first learning management system to be widely adopted. Blackboard Inc eventually bought WebCT, and UBC still uses Blackboard Connect as its LMS.

In the early 2000s,  a ‘nascent’ governance structure for learning technologies was suggested, and in recent years governance has focused mainly on the transition from Blackboard Vista to Blackboard Connect. However, over the last couple of years, UBC has also developed a major flexible learning strategy which is now being extensively implemented throughout the university. There has been considerable frustration and dissatisfaction with the implementation of Connect which has been getting in the way of the flexible learning strategy, so I see this report as a way of fixing that disconnect (sorry for the pun.) Or, as the report puts it:

Faculty desire a greater choice of tools, so that the one with the best fit for the pedagogical purpose can be selected….the functional footprint of the LMS is shrinking over time though the footprint of the entire [LT] ecosystem is arguably increasing. We anticipate a shrinking LMS footprint while still envisaging the need for a core within the ecosystem.

Comment

Although specific to UBC, this report will resonate with many other institutions. It should be essential reading for any Provost concerned with moving their institution forward into digital learning, as institutions struggle with legacy technology systems. The report adopts a clear, evidence-based analytical approach to sensitive issues around management, technology choice, and pedagogy, even if occasionally the business-speak language grates a little.

So back to my glass of Prosecco on the sun-drenched terrace.