November 18, 2017

Answering questions about teaching online: assessment and evaluation

How to assess students online: remote exam proctoring

How to assess students online: remote exam proctoring (see Chapter 4.5 on competency-based learning)

Following on from my Contact North webinar on the first five chapters of my book, Teaching in a Digital Age, and my blog post on this yesterday, there were four follow-up questions from the seminar to which I posted written answers. Here they are:

Unanswered Questions

Q: ­If multiple choice is not great for applied learning assessment – could you please give us some tips for more effective assessment in the virtual environment?­

Big question! There are several ways to assess applied learning, and their appropriateness will depend on the subject area and the learning goals (Look particularly at Appendix A, Section 8). Here are some examples:

  • via project work, where the outcome of the project is assessed. (This could be either an individual or group project). Marking a project that may take several weeks work on the part of students helps keep the marking workload down, although this may be offset to some extent by the help that may need to be given to learners during the project.
  • through e-portfolios, where students are asked to apply what they are learning to practical real-life contexts. The e-portfolio is then used to assess what students have learned by the end of the course.
  • use of online discussion forums, where students are assessed on their contributions, in particular on their ability to apply knowledge to specific real world situations (e.g. in contemporary international politics)
  • using simulations where students have to input data to solve problems, and make decisions. The simulation collects the data and allows for qualitative assessment by the instructor. (This depends on there being suitable simulations available or the ability to create one.)

Q: I am finding in my post-graduate online courses the professor is interacting less and less in the online weekly forums, while I know there are competing theories as to how much they should interact with students, do you have an opinion on whether or not professors should or should not interact weekly? Personally, I enjoy their interaction I find it furthers my learning.

This is another big issue. In general, the research is pretty much consistent: in online learning, instructor ‘presence’ is critical to the success of many students. Look particularly at Chapter 4, Section 4 and Chapter 11, Section 10. However presence alone is not sufficient. The online discussion must be designed properly to lead to academic learning and the instructor’s intervention should be to raise the level of thinking in the discussion (see 4.4.2 in the book). Above all, the discussion topics must be relevant and from a student’s perspective clearly contribute to answering assessment questions better. The instructors should in my view be checking daily their online discussion forums and should respond or intervene at least weekly. Again though this is a design issue; the better the course design, the less they should need to log in daily.

Q: Can you give an example of how a MOOC can supplement a face-to-face or fully online course?

I think the best way is to consider a MOOC as an open educational resource (OER). There is a whole chapter (Chapter 10) in the book on OERs. Thus MOOCs (or more likely parts of MOOCs) might be used in a flipped classroom context, where students study the MOOC then come to class to do work around it. But be careful. Many MOOCs are not OER. They are protected by copyright and cannot be used without permission. They may be available only for a limited period. If it is your own MOOC, on the other hand, that’s different. My question is though: is the MOOC material the best OER material available or are there other sources that would fit the class requirement better, such as an open textbook? Or even better, should you look at designing the course completely differently, to increase student interaction, self-learning and the development of higher order thinking skills, by using one of the other teaching methods in the book?

Q: Would better learning analytics reports help teachers have a more relevant role in MOOCS?

Learning analytics can be helpful but usually they are not sufficient. Analytics provide only quantitative or measurable data, such as time on task, demographics about successful or unsuccessful students, analysis of choices in multiple-choice tests, etc. This is always useful information but will not necessarily tell you why students are struggling to understand or are not continuing. Compare this with a good online discussion forum where students can raise questions and the instructor can respond. Students’ comments, questions and discussion can provide a lot of valuable feedback about the design of the course, but require in most cases some form of qualitative analysis by the instructor. This is difficult in massive online courses and learning analytics alone will not resolve this, although they can help, for instance, in focusing down on those parts of the MOOC where students are having difficulties.

Any more questions?

I’m more than happy to post regular responses to any questions you may have about online teaching, either related to the book or quite independent of it. Just send them to me at tony.bates@ubc.ca

Is content still important in a digital age?

© handyguyspodcast.com

© handyguyspodcast.com

The story so far

Chapter 5 of my open textbook, ‘Teaching in a Digital Age’ is about the design of teaching and learning, which I am currently writing and publishing as I go.

I started Chapter 5 by suggesting that instructors should think about design through the lens of constructing a comprehensive learning environment in which teaching and learning will take place. I have started to work through the various components of a learning environment, focusing particularly on how the digital age affects the way we need to look at some of these components.

I started by looking at how the characteristics of our learners are changing. This post now examines how our perspectives on content are being influenced by the digital age. The following posts will do the same for skills, learner support, resources and assessment respectively.

This will then lead to a discussion of different models for designing teaching and learning, which attempt to provide a structure and integration of these various components of a learning environment.

Managing content

For most teachers and instructors, content remains a key focus. Content includes facts, ideas, principles, evidence, and descriptions of processes or procedures. A great deal of time is spent on discussing what content should be included in the curriculum, what needs to be covered in a course or a program, what content sources such as text-books students should access, and so on.

Teachers and instructors often feel pressured to cover the whole curriculum in the time available. In particular, lecturing or face-to-face classes remain a prime means for organizing and delivering content. I have already made a case for balancing content with skills development, but issues around content remain critically important in teaching. However, there are some developments resulting from a digital age that require special attention with regard to content.

Goals for content

Because as instructors we tend to take content for granted – this is what we teach – it is important, when designing teaching for a digital age, to be clear in our goals for teaching content. Why do we require students to know facts, ideas, principles, evidence, and descriptions of processes or procedures? Is learning specific content a goal in itself, or is it a means to an end? For instance, is there an intrinsic value in knowing the periodic table, or the dates of battles, or are they means to an end, such as designing experiments or understanding why French is an official language in Canada?

The question is important, because in a digital age, some would argue that learning or memorising content becomes less important or even irrelevant when it is easy just to look up facts or definitions or equations. Cognitivists will argue that content needs to be framed or put in context for it to have meaning. Thus it could be argued that knowing not only the dates of a battle and who won but also why the battle was fought and its significance in the later development of a country is the reason for knowing about the battle, its date and the victors. Or does content need to be learned solely to enable us to do things, such as solve problems, or make decisions, and we need only to draw on content as and when needed, since it is now so easy to access?

Probably more important than the instructor being clear on why content is being taught is for the students to understand why. One way of stating this is to ask: what value is added to the overall goals of this course or program by teaching this specific content? Do students need to memorise this content, or know where to find it, and when it is important to use it? This means of course having very clear goals for the course or program as a whole.

Quantity and depth

In many contexts, instructors have little choice over content. External bodies, such as accreditation agencies, state or provincial governments, or professional licensing boards, may well dictate what content a particular course or program needs to cover. However, the rapid growth of scientific and technological knowledge increasingly challenges the idea of a fixed body of content that students must learn. Engineering and medical programs struggle to cover even in six or eight years of formal education all the knowledge that professionals need to know to practice effectively. Professionals will need to go on learning well past graduation if they are to keep up with new developments in the field.

In particular, covering content quickly or overloading students with content are not effective teaching strategies, because even working harder all waking hours will not enable students in these subject domains to master all the information they might possibly need in their professions. Specialization has been a traditional way of handling the growth of knowledge, but that does not help in dealing with complex problems or issues in the real world, which often require inter-disciplinary and broader based approaches. Thus instructors need to develop strategies that enable students to cope with the massive and growing amounts of knowledge in their field.

One way to handle the problem of knowledge explosion is to focus on the development of skills, such as knowledge management, problem-solving and decision-making. However, these skills are not content-free. In order to solve problems or make decisions, you need access to facts, principles, ideas, concepts and data. To manage knowledge, you need to know what content is important and why, where to find it, and how to evaluate it. In particular there may be core or basic knowledge or content that needs to be mastered for many if not most of their professional activities. One teaching skill then will be the ability to differentiate between essential and desirable areas of content, and to ensure that whatever is done to develop skills, in the process core content is also covered.

Sources

Another critical decision for teachers in a digital age is where students should source or find content. In medieval times, books were scarce, and the library was an essential source of content not only for students but also for professors. Professors had to select, mediate and filter content because the sources of content were extremely scarce. We are not in that situation today. Content is literally everywhere: on the Internet, in social media, on mass media, in libraries and books, as well as in the lecture theatre.

Often, a great deal of time is spent in departmental or program meetings on discussing what textbooks or articles students should be required to read. Part of the reason for selecting or limiting content is to limit the cost to students, as well as the need to focus on a limited range of material within a course or program. But today, content is increasingly open, free and available on demand over the Internet. It has already been argued that most students will need to continue learning after graduation. They will increasingly resort to digital media for their sources of knowledge. Therefore when deciding on content we should be considering:

(a) to what extent does the instructor need to choose the content for a program (other than a broad set of curriculum topics) and to what extent should students be free to choose both content and the source of that content?

(b) to what extent does the instructor need to deliver content themselves, such as through a lecture or Powerpoint slides, when content is so freely available elsewhere? What is the added value you are providing by delivering the content yourself? Could your time be better used in other ways?

(c) to what extent do we need to provide criteria or guidelines to students for choosing and using openly accessible content, and what is the best way to do that?

When answering such questions, we should also be asking whether our decisions will help students manage content better themselves after graduating.

Structure

One of the most critical supports that teachers and instructors provide is to structure the sequence and inter-relationship of different content elements. I include within structure:

  • the selection and sequencing of content,
  • developing a particular focus or approach to specific content areas,
  • helping students with the analysis, interpretation or application of content
  • integrating and relating different content areas.

Traditionally, content has been structured by breaking a course into a number of topic-related classes delivered in a particular sequence, and within the classes, by instructors ‘framing’ and interpreting content. However, new technologies provide alternative means to structure content. Learning management systems such as Blackboard or Moodle enable instructors to select and sequence content material, which students can access anywhere, at any time – and in any order. The availability of a wide range of content over the Internet, and the ability to collect and sort content through blogs, wikis,and e-portfolios enable students increasingly to impose their own structures on content.

Students need some form of structure within content areas, partly because some things need to be learned in ‘the right order’, partly because without structure content becomes a jumble of unrelated topics, and partly because students can’t know or work out what is important and what is not within a total content domain, at least until they have started studying it. Novice students in particular need to know what they must study each week. There is a good deal of research evidence to suggest that novice students benefit a great deal from tightly structured, sequential approaches to content, but as they become more knowledgeable or experienced in the domain, they seek to develop their own approaches to the selection, ordering and interpretation of content.

Therefore in deciding on the structure of the content in a course or program instructors need to ask:

(a) how much structure should I provide in managing content, and how much should I leave to the students?

(b) how do new technologies affect the way I should structure the content? Will they enable me to provide more flexible structures that will suit a diverse range of student needs?

Similarly, when answering these questions we should ask how important it is for students themselves to be able to structure content, and whether our answers to the two questions above will further help them to do this.

Activities

Lastly, what activities do we need to ask students to do to help them learn content? To answer this question will mean returning to the goals for learning content and the overall goals of the course:

  • if memorization is important, then automated tests such as computer-marked assignments with correct answers being provided can be used
  • if the aim is to enable students to draw on content such as facts, principles, data or evidence to construct an argument, to solve equations, or to design an experiment, then opportunities for practising such skills that require students to draw on specific content will be needed
  • if the aim is to help students to manage knowledge, then we may need to set tasks for students to do that requires them to select, evaluate, analyse and apply content.

We shall see that technology enables us to widen considerably the range of activities that students can use to master content, but these need to be related to the learning goals set for the course of program. Without a planned set of activities, though, content may just enter the brain one day and leave it the next.

In conclusion

Even or especially in a digital age, content, in terms of things to know, remains critically important, but in a digital age the role of content is subtly changing, in some ways becoming a means to other ends, such as skills development, rather than an end in itself. Because of the rapid growth in knowledge in nearly all subject areas, being clear about the role and purpose of content in a course, and communicating that effectively to students, becomes particularly important.

New technologies offer alternative ways to present and structure content and raise the fundamental question of the role of teachers or instructors in delivering content when so many alternative sources are available. Lastly in any modern learning environment, activities need to be designed that enable students to master and appropriately apply essential content.

Over to you

So wha’d’ya think?

  • Is this a useful way at looking at content in a digital age or is it all pretty obvious?
  • is the distinction between content and skills useful, or is it a false dichotomy?
  • do you think there is too much emphasis on learning content and not enough on skills development?
  • does the increased accessibility of different sources of content change the game – or should instructors continue to select and control access to content for students?

Love to hear from you.

Next up

Developing skills for a digital age