November 18, 2017

What do we mean by quality when teaching in a digital age?

© Insights, 2012

© Insights, 2012

Before I start on my nine steps to quality learning for my open textbook, Teaching in a Digital Age, I have needed to ‘clear the decks’ about what we mean by quality. I thought this bit might be useful to share, as quality is a very slippery concept at the best of times.

The aim of this chapter is to provide some practical guidelines for teachers and instructors to ensure quality teaching in a digital age. Before I can do this, however, it is necessary to clarify what is meant by ‘quality’ in education, because I am using ‘quality’ here in a very specific way.

Definitions

Probably there is no other topic in education which generates so much discussion and controversy as ‘quality’. Many books have been written on the topic, but I will cut to the chase and give my definition of quality up-front. For the purposes of this book, quality is defined as:

teaching methods that successfully help learners develop the knowledge and skills they will require in a digital age.

This of course is the short answer to the question. A longer answer means looking, at least briefly, at:

  • institutional and degree accreditation
  • internal (academic) quality assurance processes
  • differences in quality assurance between traditional classroom teaching and online and distance education
  • the relationship between quality assurance processes and learning outcomes
  • ‘quality assurance fit for purpose’: meeting the goals of education in a digital age.

This will then provide the foundations for my recommendations for quality teaching that will follow in this chapter.

Institutional and degree accreditation

Most governments act to protect consumers in the education market by ensuring that institutions are properly accredited and the qualifications they award are valid and are recognised as of being of ‘quality.’ However, the manner in which institutions and degrees are accredited varies a great deal. The main difference is between the USA and virtually any other country.

The U.S. Department of Education’s Network for Education Information states in its description of accreditation and quality assurance in the USA:

Accreditation is the process used in U.S. education to ensure that schools, postsecondary institutions, and other education providers meet, and maintain, minimum standards of quality and integrity regarding academics, administration, and related services. It is a voluntary process based on the principle of academic self-governance. Schools, postsecondary institutions and programs (faculties) within institutions participate in accreditation. The entities which conduct accreditation are associations comprised of institutions and academic specialists in specific subjects, who establish and enforce standards of membership and procedures for conducting the accreditation process.

Both the federal and state governments recognize accreditation as the mechanism by which institutional and programmatic legitimacy are ensured. In international terms, accreditation by a recognized accrediting authority is accepted as the U.S. equivalent of other countries’ ministerial recognition of institutions belonging to national education systems.

In other words, in the USA, accreditation and quality assurance is effectively self-regulated by the educational institutions and faculty through their control of accreditation agencies, although the government does have some ‘weapons of enforcement’, mainly through the withdrawal of student financial aid for students at any institution that the U.S. Department of Education deems to be failing to meet standards.

In many other countries, government has the ultimate authority to accredit institutions and approve degrees, although in countries such as Canada and the United Kingdom, this is often exercised by arm’s length agencies appointed by government, but consisting mainly of representatives from the various institutions within the system. These bodies have a variety of names, but Degree Quality Assurance Board is a typical title.

However, more important than the formal lines of responsibility for quality is how the accrediting agencies actually operate. Usually, once a degree program is approved, there is little follow-up or monitoring afterwards, unless formal complaints are subsequently made about the quality of the program, although many institutions now voluntarily review programs every five years or so. Also, once an institution has been accredited, the accreditation agency may delegate back to the institution the approval of it own degree programs, providing that it has an internal process in place for assuring quality, although where government is formally responsible, new degrees may still come to an accrediting agency, to ensure there is no duplication within the system, that there is a defined market for the program, or where approval to deviate from government guidelines on fees may be requested. Nevertheless, mainly to ensure academic freedom from direct government interference, universities in particular have a large degree of autonomy in most economically advanced countries for determining ‘quality’ in programming.

However, in recent years, some regulatory agencies such as the United Kingdom’s Quality Assurance Agency for Higher Education have adopted formal quality assurance processes based on practices that originated in industry. The U.K. QAA’s Quality Code for Higher Education which aims to guide universities on what the QAA is looking for runs to several hundred pages. Chapter B3 on Learning and Teaching is 25 pages long and has seven indicators of quality. Indicator 4 is typical:

Higher education providers assure themselves that everyone involved in teaching or supporting student learning is appropriately qualified, supported and developed.

Many institutions as a result of pressure from external agencies have therefore put in place formal quality assurance processes over and beyond the normal academic approval processes (see Clarke-Okah et al., 2014, for a typical, low-cost example).

Internal quality assurance

It can be seen then that the internal processes for ensuring quality programs within an institution are particularly important. Although again the process can vary considerably between institutions, at least in universities the process is fairly standard. A proposal for a new degree will usually originate from a group of faculty/instructors within a department. The proposal will be discussed and amended at departmental and/or Faculty meetings, then once approved will go to the university senate for final approval. The administration in the form of the Provost’s Office will usually be involved, particularly where resources, such as new appointments, are required.

Although this is probably an over-generalisation, significantly the proposal will contain information about who will teach the course and their qualifications to teach it, the content to be covered within the program (often as a list of courses with short descriptions), a set of required readings, and usually something about how students will be assessed. Increasingly, such proposals may also include broad learning outcomes for the program.

If there is a proposal for courses within a program or the whole program to be delivered fully online, it is likely that the proposal will come under intense internal scrutiny. What is unlikely to be included in a proposal though is what methods of teaching will be used. This is usually considered the responsibility of individual faculty members. It is this aspect of quality with which this chapter is concerned.

Lastly, some institutions require every program to be reviewed after five or more years of operation, or at the discretion of the senior administration. Again whether and how this is done varies considerably. One common approach is for an internal review process, with an internal evaluation report by a committee set up by the department offering the program, followed by a review of the internal committee’s report by external assessors. This review may or (more frequently) may not lead to significant changes in a program, but this will depend on the instructors responsible for teaching the program agreeing to implement any recommended changes. Less frequently, where enrolment for a program has declined to unacceptable levels or where external complaints about a program have been received, the Vice President Academic may call for an external review of the program, in which case anything is possible, up to and including closure of the program.

Jung and Latchem (2102), in a review of quality assessment processes in a large number of online and distance education institutions around the world, make the following important points about quality assurance processes within institutions:

  • focus on outcomes as the leading measure of quality
  • take a systemic approach to quality assurance
  • see QA as a process of continuous improvement
  • move the institution from external controls to an internal culture of quality
  • poor quality has very high costs so investment in quality is worthwhile.

In particular, Butcher and Wilson-Strydom (2013) warn:

you should not assume that creating quality assurance structures… automatically improves quality….Institutional quality assurance structures and processes are important, but beware of making them an exercise in compliance for accountability, rather than a process of learning and self-improvement that really improves quality.

There are many guidelines for quality traditional classroom teaching. Perhaps the most well know are those of Chickering and Gamson (1987), based on an analysis of 50 years of research into best practices in teaching. They argue that good practice in undergraduate education:

  1. Encourages contact between students and faculty
  2. Develops reciprocity and cooperation among students.
  3. Encourages active learning.
  4. Gives prompt feedback.
  5. Emphasizes time on task.
  6. Communicates high expectations.
  7. Respects diverse talents and ways of learning.

Online courses and programs

Because online learning was new and hence open to concern about its quality, there have also been many guidelines, best practices and quality assurance criteria created and applied to online programming. All these guidelines and procedures have been derived from the experience of previously successful online programs, best practices in teaching and learning, and research and evaluation of online teaching and learning.

Some degree quality assurance boards (such as the QAA in the UK and PEQAB in Ontario) have put in place specific ‘benchmarks’ for online courses. A comprehensive list of online quality assurance standards, organizations and research on online learning can be found in Appendix 3. Graham et al. (2001) applied Chickering and Gamson’s seven principles for face-to-face teaching to the evaluation of four online courses from a mid-western university in the USA, and adapted these principles for online learning.

Thus ensuring quality in online learning is not rocket science. There is plenty of evidence of what works and what doesn’t, which will be examined in more detail in this chapter. There is no need to build a bureaucracy around this, but there does need to be some mechanism, some way of calling institutions when they fail to meet these standards. However, we should also do the same for campus-based teaching. As more and more already accredited (and ‘high quality’) campus-based institutions start moving into hybrid learning, the establishment of quality in the online learning elements of programs will become even more important.

Thus there are plenty of evidence-based guidelines for ensuring quality in teaching, both face-to-face and online. The main challenge then is to ensure that teachers and instructors are aware of these best practices and that institutions have processes in place to ensure that guidelines for quality teaching are implemented and followed.

Quality assurance, innovation and learning outcomes

It may have been noted that most QA processes are front-loaded, i.e. they look at inputs – such as the academic qualifications of faculty, or the processes to be adopted for effective teaching, such as clear learning objectives – rather than outputs, such as what students have actually learned. They also tend to be backward-looking, that is, they focus on past best practices.

This is particularly important for evaluating new teaching approaches. Butcher and Hoosen (2014) state:

The quality assurance of post-traditional higher education is not straightforward, because openness and flexibility are primary characteristics of these new approaches, whereas traditional approaches to quality assurance were designed for teaching and learning within more tightly structured frameworks.

 However, Butcher and Hoosen (2014) go on to say that:

fundamental judgements about quality should not depend on whether education is provided in a traditional or post-traditional manner …the growth of openness is unlikely to demand major changes to quality assurance practices in institutions. The principles of good quality higher education have not changed…. Quality distance education is a sub-set of quality education…Distance education should be subject to the same quality assurance mechanisms as education generally.’

Such arguments though offer a particular challenge for teaching in a digital age, where it is argued that learning outcomes need to include the development of skills such as independent learning, facility in using social media for communication, and knowledge management, skills that have not been explicitly identified in the past. Quality assurance processes are not usually tied to specific types of learning outcomes, but are more closely linked to general performance measures such as course completion rates, time to degree completion and grades based on past learning goals.

Furthermore, we have already seen in Chapters 9 and 10 that new media and new methods of teaching are emerging that have not been around long enough to be subject to analysis of best practices. A too rigid view of quality assessment based on past practices could have serious negative implications for innovation in teaching and for meeting newly emerging learning needs. ‘Best practice’ may need occasionally to be challenged, so new approaches can be experimented with and evaluated.

Quality assurance: fit for purpose in a digital age

Maxim Jean-Louis, the President of Contact North, at a presentation in 2010 to the Higher Education Quality Council of Ontario, made a very useful distinction about different ways of looking at quality in education:

  • Quality as ‘Excellence’– a definition that sets abstract goals for institutions and academic communities to always striving to be the best, mainly taken as having elitist undertones. In post-secondary education this could mean winning Nobel prizes, attraction of research funds or the “best” faculty as measured by research output and teaching evaluations. The drawback here is that this tends to also exclude the work of the ‘further education’ sectors, and is not applied equally between disciplines (citation counts do not exist for historians and many other subjects).
  • Quality as ‘Meeting a pre-determined standard’– a definition that requires only a given standard to be met, e.g. a minimum grade, basic competency, the ability to read, write, use a computer, etc. [It might also include competencies and skills, degree completion rates, time to degree completion, etc.] The drawback of this is that setting and measuring this ‘standard’ is difficult at best and idealistic at worst.
  • Quality as ‘fitness for purpose’ – in this construction of quality, we have to decide the extent to which the service or product meets the goals set – does this course or program do what it says it was going to do? Such a construction of quality allows institutions/sectors to define goals themselves according to their mandate and concentrates on meeting the needs of their customers (whether this be upgrading learners, graduate researchers, industry, etc.).

Quality assurance processes must address the increasing diversity of our educational systems. Distance education organizations are not the same as elite traditional universities and shouldn’t try to be. This would mean looking for different measures of quality in the Open University, for instance, than in Cambridge University. Neither one is necessarily better (depending on what they are trying to achieve), but the learning experience ought to be different, even though the intended learning outcomes may be similar; this will mean different design criteria but not necessarily different criteria for assessing the quality of the learning.

In the meantime, much more attention needs to be directed at what campus-based institutions are doing when they move to hybrid or online learning. Are they following best practices, or even better, developing innovative, better teaching methods? The design of xMOOCs and the high drop-out rates of many new two year colleges new to online learning in the USA suggest they are not.

This means that different types of institution will and should evaluate quality differently. If the goal or purpose is to develop the knowledge and skills that learners will need in a digital age, then this is the ‘standard’ by which quality should be assessed, while at the same time taking into account what we already know about general best practices in teaching. The recommendations for quality teaching in a digital age that follow in this chapter are based on these principles.

Over to you

There is so much I wanted to write here about the stupidity of the current system of institutional accreditation and internal quality assurance processes, especially but not exclusively in the United Kingdom, but this section is meant as an introduction to practical guidelines for teaching and learning. So I’ve tried to be uncharacteristically restrained in writing this section. But feedback is even more welcome than usual.

1. (a) First, are there any incorrect facts in this section? This is a large and complex topic and it is easy to get things wrong.

(b) Have I left out anything really important about assessing quality in teaching and learning?

2. One problem with this topic is that it tends to gravitate between two polarised positions: those who believe in absolute truth and those who are relativists. Absolute truthers believe that there is a God-given set of ‘quality’ standards that are set primarily by elite institutions that everyone else should strive to meet. Relativists (like myself) believe that quality is in the eye of the beholder; it all depends on what your goals are. Hence my definition of quality is set among the rather limited goal in one way – and extremely ambitious in another – of developing teaching methods that will help learners develop the knowledge and skills they will need in a digital age. So: any views on my definition of quality? Is it fit for purpose?

3. What do you think of the current system of (a) institutional accreditation and (b) internal quality assurance processes?

My own view is that institutional accreditation is definitely needed to protect against really incompetent or downright dishonest organisations, but, depending on the jurisdiction, it is very much an insider’s process and not very transparent, and while current accreditation processes may set minimum standards it certainly doesn’t do much to improve quality in the system.

Similarly, internal quality assurance processes are far too cosy and protect the status quo. The internal program approval processes are based again on peer review of a very limited kind, with often ‘I’ll scratch your back if you’ll scratch mine’ approach to program approval. I’ve been on a number of program reviews as an external reviewer, but rarely see any significant changes, despite sometimes scathing reviews from the externals.

And as for formal QA processes, they are the kiss of death for quality, tangling faculty and administrators in incredibly bureaucratic processes without dealing with the real issues around quality teaching and learning.

Of course, all these practices are in the name of protecting academic freedom, which is important – but surely better processes can be derived for improving quality without threatening academic freedom.

4. So lastly, is it wise for me to restrain myself from adding these types of comments in the book – or will I muddy the waters of what is to come if I do?

References and further reading

Butcher, N. and Wilson-Strydom, M. (2013) A Guide to Quality in Online Learning Dallas TX: Academic Partnerships

Butcher, N. and Hoosen, S. (2014) A Guide to Quality in Post-traditional Online Higher Education Dallas TX: Academic Partnerships

Chickering, A., and Gamson, Z. (1987) ‘Seven Principles for Good Practice in Undergraduate Education’ AAHE Bulletin, March 1987.

Clarke-Okah, W. et al. (2014) The Commonwealth of Learning Review and Improvement Model for Higher Education Institutions Vancouver BC: Commonwealth of Learning

Graham, C. et al. (2001) Seven Principles of Effective Teaching: A Practical Lens for Evaluating Online Courses The Technology Source, March/April

Jung, I. and Latchem, C. (2012) Quality Assurance and Accreditation in Distance Education and e-Learning New York/London: Routledge

Comments

  1. Kevin Forgard says:

    Tony,
    First, I have to admit that I do not make enough time to read your blog, so hopefully my comment is not too out of context.
    I have been an instructional designer in higher ed for about 5 years and have also found quality assurance tools to be a little too constraining. The type of work that I do is too diverse and requires flexibility when working with faculty – particularly those who are not really sure what an instructional designer does, or worse, sees the profession as a threat to their academic freedom.
    As you mention in the QA section, a set of standards needs to be flexible. Constructs such as Quality Matters are useful, but are not absolute. They provide a common language and framework, but should not be followed verbatim. Learning is messy, so its design should be flexible, but it is not completely chaotic.

    Thanks for opening debate on this topic.
    Like a wonderful history professor I worked with in Massachusetts, who every time a colleague brought up “best practices”, she would call them out, “best for who?”

  2. Harvey Mellar says:

    Small factual detail, you write

    “Some degree quality assurance boards (such as the QAA in the UK and PEQAB in Ontario) have put in place specific ‘benchmarks’ for online courses”.

    My understanding is that the QAA Quality Code no longer has specific benchmarks for online courses – I think you are referring to the 1999/2004 Code of Practice which you list in the resources for Appendix 3, which has now been replaced by the 2014 Quality Code.

    This change in the Code is significant in that it signals a move away from specific criteria for online courses, towards embedding discussion of online provision across the whole Code.

    • Many thanks for that correction, Harvey. You are right, I did base this sentence on the former code of practice. However, my other comment on the indicators was taken from the more recent code.
      I also agree: it’s a good move as online learning should be treated as just one method of several and all should meet same the standards of quality

  3. Hello Tony! I really enjoy reading your work: our minds seem to function in the same ways. I really appreciated your comments about trying to “be uncharacteristically restrained.” 🙂 I’ve fought that necessity all my life…which brings me to the eQcheck eLearning quality certification system that, after 13 years, remains futuristic to this day. Perriodically I think about reviving it because, as you say, most (if not all) eLearning quality assurance systems come from the provider perspective and typically describe current practice as “the standard”. Worse yet, they use peer review and we share the view on why that doesn’t work.

    I’m in Qatar now, writing policy as Higher Education Expert for the Supreme Education Council. We’re looking at how to assure the quality of online degrees for the purposes of scholarships and degree attestation. I’ve been away from the eL QA field for a bit, so I need you to confirm my conclusion that (1) there is no recognized international accreditation service and (2) there is a need for one. What do you think?

    Very best regards, Kathryn

    • Hi, Kathryn

      Good to hear from you. As you probably know, the Commonwealth of Learning has an excellent site on quality assurance and lists several regional quality assurance initiatives. However, here is not one listed specifically for the Middle East.
      EFQUEL (the European Foundation for Quality in e-Learning) offers UNIQUe, a high quality institutional certification for outstanding use of ICT in learning and teaching. It is awarded to universities or institutes, after a process of self-assessment and external peer review, for renewable periods of three years. I don’t know though whether it will do this outside Europe (probably.)
      In any case, both the CoL and EFQUEL models are primarily internal quality assurance processes, so there is no international accreditation agency I know of that will officially ‘recognise’ these. Thus formal national accreditation agencies are still the gold standard for recognition.
      Good luck in Qatar. If you can get me World Cup soccer tickets, let me know!

  4. A nice and timely review, Tony! I was surprised that you did not mention some of the long established rubrics for quality, such as Quality Matters or the Chico State Rubric for Online Instruction. These of course are focused on course design, not necessarily excellence, standards, or fitness for purpose.

    A philosophical question is what do we mean by quality if the focus shifts from the “teaching” to the “learning”?

    • Hi, Britt

      Thanks for your comment. Quality Matters is listed in E-learning quality assurance standards, organizations and research, which is available in full in Appendix 3 of the book.
      My problem with rubrics is, as I mentioned in the post, that they are things to do before designing a course rather than assessing the quality of the course itself. In particular it would be difficult if not impossible to prove that any particular rubric leads to better learning outcomes. As a check list, rubrics are useful guidelines (and Quality Matters is one of the most comprehensive) but they are not sufficient on their own to ensure high quality learning, because they miss important processes that happen during the teaching and learning, such as the relationship of an instructor with the students.
      Lastly if these rubrics are so important for online learning, why are there not similar rubrics for face-to-face teaching? Double standards are being applied here. As online and face-to-face teaching become more integrated we need an approach to quality that works for all kinds of teaching, not just fully online learning – but I sense you agree with me on that!

  5. Blessings & Greeting Mr. Bates!

    Thank you for all the authentic intrinsic work that you do and for reaching out selflessly to this wonderful planet of ours, enlightening others and truly assisting all for a better outcome! You never know how greatly your efforts are affecting others positively, especially when they are done out of good well consciousness and in a tangible manner.

    I think it is important for us all to realize that everything is reciprocal in life; your servitude is very much appreciated 🙂 The fact that your website is available for anyone is truly a blessing and a positive contribution to the world because it is based on progressive facts. Your demeanour is genuine, so thank you for your talents, sharing your resourceful website, and for being you! This is my first time on your website by the way, but I was online 3 days ago, searching for free online education, your name came up on some website, the way you responded to people made me note your name 🙂

    My questions are:

    Who were and are your mentors?

    Which free online courses and schools do you recommend for someone who is interested in a variety of business, neuroscience, and humanitarian work?

    What is your opinion on all these technical schools that condense such programs as Dental Hygiene and charge a whopping 40 grand for 18 months of study at their government accredited venues?

    *That someone is me 🙂 I’m starving for education, also in need to expand my horizons when it comes to profitable employment that consists of work/life balance. I have post -secondary and technical schooling and lots of work experience where obviously I have adapted many skills. Being a single mother of 2 amazing kids, I am currently looking for the best free E-Schooling options out there that are considered viable. I acquired the IT Management Diploma yesterday from ALISON; I enrolled 2 days before that.

    Thank you kindly for taking the time to read this, and kudos for all your hard work! Wishing you a fantastic week end!

    Kind Regards,
    Mirna H
    Ottawa, Canada

    • Thank you for your kind comments, which are much appreciated, and I apologize for the delay in replying.

      You ask: Which free online courses and schools do you recommend for someone who is interested in a variety of business, neuroscience, and humanitarian work?

      That is a very hard question to answer, but the simplest answer is probably ‘none.’ There is always a cost of some kind, and certainly in Ontario you would best be advised to take a course from any of the many good provincial government supported colleges or universities and try to find some kind of financial support in the form of grants or loans. Algonquin College and the University of Ottawa or Carleton University all offer courses that lead to professional qualifications in these areas.

      See also my earlier answer to Latecia on my blog post.

      Would other readers like to offer other advice to Mirna and Latecia?

      Regards

      Good luck with your future studies,

      regards

Speak Your Mind

*