December 20, 2014

EDEN research papers: OERs (inc. MOOCs), quality/assessment, social media, analytics and research methods

Listen with webReader

EDEN RSW me 2

EDEN has now published a second report on my review of papers submitted to the EDEN research workshop in Oxford a couple of weeks ago. All the full papers for the workshop can be accessed here.

Main lessons (or unanswered questions) I took away:

OERs and MOOCs

  • what does awarding badges of certificates for MOOCs or other OER actually mean? For instance will institutions give course exemption or credits for the awards, or accept such awards for admission purposes? Or will the focus be on employer recognition? How will participants who are awarded badges know what their ‘currency’ is worth?
  • can MOOCs be designed to go beyond comprehension or networking to develop other critical 21st century skills such as critical thinking, analysis and evaluation? Can they lead to ‘transformational learning’ as identified by Kumar and Arnold (see Quality and Assessment below)
  • are there better design models for open courses than MOOCs as currently structured? If so what would they look like?
  • is there a future for learning object repositories when nearly all academic content becomes open and online?

Quality and assessment

  • research may inform but won’t resolve policy issues
  • quality is never ‘objective’ but is value-driven
  • the level of intervention must be long and significant enough to result in significant learning gains
  • there’s lots of research already that indicates the necessary conditions for successful use of online discussion forums but if these conditions are not present then learning will not take place
  • the OU’s traditional model of course design constrains the development of successful collaborative online learning.

Use of social media in open and distance learning

There were surprisingly few papers on this topic. My main takeaway:

  • the use of social media needs to be driven by sound pedagogical theory that takes into account the affordances of social media (as in Sorensen’s study described in an earlier post under course design)

Data analytics and student drop-out

  • institutions/registrars must pay attention to how student data is tagged/labeled for analytic purposes, so there is consistency in definitions, aggregation and interpretation;
  • when developing or applying an analytics software program, consideration needs to be given to the level of analysis and what potential users of the data are looking for; this means working with instructional designers, faculty and administrators from the beginning
  • analytics need to be integrated with action plans to identify and support early at risk students

Research methods

Next

If these bullets interest you at all, then I strongly recommend you go and read the original papers in full – click here. My summary is of necessity personal and abbreviated and the papers provide much greater richness of context.

 

 

Developing a coherent approach to mobile learning

Listen with webReader
The JIBC Emergency Social Services app

The JIBC Emergency Social Services app

The Justice Institute of British Columbia (JIBC) is an unusual organization, focusing on the training of police, fire, corrections and paramedical personnel, as well as providing training for social service departments of the BC provincial government.

Tannis Morgan, Associate Dean, JIBC Centre for Teaching, Learning and Innovation, has just posted a very interesting post, Mobile Learning at an Applied Institution, on why and how JIBC has developed its mobile learning strategy, which includes issuing pre-loaded iPads to participants in some programs. If you have any interest in mobile learning, the post is well worth reading. You should click on the graphic at the end of the post under ‘Presentation for ETUG 2014′ for more information from slides prepared for an ETUG workshop.

Particularly worth noting is that most of JIBC’s mobile applications are open and free, because the materials are of direct value to all personnel working in this area, whether they are taking a JIBC course or not. Making the material developed for clients such as the government’s emergency social services department open and free has resulted in more enrolments, as employees and managers see the value of the service that JIBC is providing.

Writing an open textbook: a mid-term report on the technology

Listen with webReader

Open textbooks free 2

I’m about half-way through writing my open textbook, ‘Teaching in a Digital Age.’ I’ve done about five and a half chapters, and I would like to share my views on the underlying technology that I am using, because, while it does the job reasonably well, we are clearly in the Version 1.0 stage of software development, from an author’s perspective. I believe there is a major opportunity to develop a software authoring framework that fully exploits the open characteristics of a textbook, but we are not there yet.

Background

I’m writing this book more or less on my own, although I do have some support from an instructional designer and I’m anticipating getting some help with marketing once the book is complete. I’m also getting a lot of useful feedback, because I am publishing as the book is being developed (the first five chapters are already available here) and also publishing excerpts in this blog.

My main technical support is coming from BCcampus, which is managing a large open textbook project on behalf of the British Columbia provincial government. My book is not directly related to the provincial government-funded project, which at this stage is focused primarily on converting existing print textbooks to open, online versions. However, as the project advances, more open textbooks will need to be written from scratch. (For more on the BCcampus open textbook project project, see here.)

BCcampus has taken an ‘off-the-shelf’ open source authoring software ‘shell’ called Pressbooks, which in itself is based on WordPress. BCcampus has made some further adaptations to Pressbooks for the open textbooks that BCcampus is helping to develop. I have used the BCcampus version of Pressbooks to create my own textbook. However, anyone can use Pressbooks for free, if they wish to write an openly published book.

What I am trying to do

My goals are two-fold:

  • to openly publish a textbook on teaching in a digital age, aimed at teachers, instructors and faculty.
  • to explore ways to incorporate best teaching practice and an open education philosophy within the design of the book.

This is a report on where I’ve got to so far in authoring the book, using the Pressbooks/BCcampus template, and in particular on what I’m finding regarding the potential and limitations of the software for authoring an open textbook.

What works

It is extremely easy to start authoring with Pressbooks. After you log in to the Pressbooks main page, you can easily set up an account which is password protected. Once you have an account, you will be assigned a url which will take you to your admin page, from where you can author your book.

Anyone who has used WordPress for blogging will have no difficulty whatsoever in getting started in Pressbooks. If you already have a structure for the book in your mind, and know what you want to write, you can be writing within less than ten minutes of signing up with Pressbooks. You can also open accounts for others, such as co-authors, an editor, or an instructional designer, with password-protected access to the editing part.

Pressbooks allows you to work in private or to publish each chapter or section when ready. You can ‘export’ , in several versions, such as ePub, pdf or html, for free downloading. BCcampus is also making available, at cost, printed versions of their textbooks. The ‘exported’ version looks clean and replicates almost exactly the edited version, with embedded urls, diagrams, headings and indentation. The variety of exported formats enables use of the textbooks on various mobile devices and tablets. If the recommended technological structure is followed when writing and editing, the reader can easily navigate through the book in a variety of ways.

Thus, for basic book writing and publishing, Pressbooks is easy to use, comprehensive in the devices it can be used on, and pleasant to read.

Challenges

From the perspective however of an open textbook, I found the following challenges:

Lack of interactivity

Those of you used to using a learning management system are likely to be frustrated by the lack in Pressbooks of common features found within an LMS, such as ways to provide feedback on exercises, places where readers/students can add their own contributions, or places where monitored and edited discussions can take place. Thus some of the key opportunities to make a book more interactive and open are currently not available, without going outside the Pressbooks environment. There are two reasons for this.

1. Pressbooks was originally designed for supporting fiction writers, and as such works perfectly for them (providing they can manage to write easily in WordPress). If you want a straight read through a book, it is perfect, but this is not what you necessarily want with an educational textbook.

2. BCcampus has added some useful features, such as widgets that allow you to insert text boxes for learning objectives, student exercises, and key take-aways, but has had to disable the comment feature because the textbooks are likely to be used by many instructors with different classes. BCcampus is rightly worried that it would be confusing and overwhelming for multiple instructors if students across all the classes shared the same comment boxes. However, as an author, I want to integrate both the activities and the student responses to the activities, and above all I want comments and feedback on what I’ve written.

There are in fact really several distinct stages or uses of an open textbook:

  • book creation (which I am going through now), where feedback is needed by the author. At this stage, the comment feature is really essential. Ideally, it should be at the end of each chapter and part.
  • response from individual readers once the book is completed. I’m already getting these, as I’m publishing as I go. At least in the early days, feedback is again essential, and it would be quite manageable for the author to monitor the comments at this stage. However, over time, adoption by instructors, accumulated spam, and repetitious comments may lead the author to want to disable this feature.
  • adoption as part of a course. At this stage the comment feature needs to be disabled (or cleared), and replaced probably by a course web site, wiki or discussion forum linked specifically to a particular instructor and their course.

What I’d really like is a widget where I can just drop in a comment box in the right place, and the ability as an author to open, clear or disable it, as well as monitoring and where necessary editing it. It could be switched to open or private.

I have also explored some possible open source discussion forums or wikis, and computer-based test services, but these would have to sit outside the textbook, and I haven’t found a satisfactory service yet (although I haven’t looked very hard – suggestions welcome.)

The technological structure of the book

Unlike many online books that you will find on Kindle or iPads, Pressbooks does not output in discrete pages. The way it manages the structure of the book to enable fluent navigation by the reader is not immediately transparent to an author writing a book.

The two key features are Parts and Chapters. I assumed (incorrectly) that Parts were sub-units or sections of Chapters. This suited me, as I’m expecting a diverse audience with a wide range of prior knowledge. I assumed that many would not want to read a whole chapter on say design models, but may have a particular interest in some of the models and not in others. However, I made the basic mistake of not reading the BCcampus Authors’ Manual carefully before starting (and when I did read it, I did not understand it.) What I hadn’t realised was that Chapters link to Parts and the Parts are not intended to have much, if any, content.

Parts are really an introduction to the substance, a kind of organiser for the actual following content, which take place in the Chapters. Think of a novel: Part 1: 1969, Chapter 1: Boy meets girl. However, I rushed off and wrote Parts like sections of a chapter then cut and pasted each Part into a Chapter. I got half-way through writing the book before realising this was a mistake, thanks to a very helpful recent meeting with staff from BCcampus.

So I have ended up using a Part like an advance organiser for a chapter, and the Chapter feature for each section of a ‘Part’. This works well now, the navigation is much better, and it avoids the reader having to scroll down through an 8,000 word chapter. Some ‘Chapters’ in Pressbooks terminology are only a couple of paragraphs long and I have renamed them sections, with the Part containing the Chapter name. I also use the Part to state the purpose of the Chapter, what is covered in the chapter, and the key takeaways.

However, as you can see, the Pressbooks terminology of Parts and Chapters is really misleading. Worse, I spent two whole days cutting and repasting content I had already written in order to get the content into the right technological structure required by the software.

No mark-up facility

Unlike Word, an editor or a co-author cannot mark up drafts in Pressbooks (or WordPress for that matter – if there is a plug-in for this, please let me know.) This makes co-production of a book and getting feedback much more frustrating, especially as there is no comment feature.

If you are writing a co-edited or co-authored book, this is a major limitation, and a better strategy might be to initially edit in Google Docs or Word, then transfer everything when finished into Pressbooks or another publishing software shell. Even then, this is not a good solution because of the high risk of losing material during the transfer – and in any case, when is an open textbook ever finished? It should be a work in continuous updating.

Even for a single author, though, the inability to mark up drafts in Pressbooks is a considerable nuisance, especially if the comment feature is disabled. Not only my instructional designer, but also several readers who are following the development of the book, are copying sections from the Pressbook version into Word, marking up suggested corrections in Word, sending me the Word document, which I then go through then make any necessary changes in the Pressbooks version.

What is needed of course is a mark-up plug-in for WordPress, which would have much wider value than just open textbook authoring.

Limitations of WordPress

Some of these limitations are also limitations of writing and editing in WordPress. The feature for creating tables is so difficult to use that it is essentially useless. Some of the formatting doesn’t transfer when cutting and pasting to another screen page (which I have to do often), such as text alignment. I spend an enormous amount of time scrolling up to the top of the page, looking for the toolbox menu, to add urls, italics, lists, or indents, sometimes accidentally transferring out of the editing page and thus losing some of the more recent writing. (Apparently, in the new version of WordPress 4, the scrolling issue to get to the toolbar will be resolved – the toolbar will stay at the top of the screen, however far down you scroll).

However, I am spending far too much time on editing and not enough on creative writing. Editing is always a time-consuming but necessary activity when writing, but I really could do without technology frustrations when editing.

Conclusions

Pressbooks is a workable solution for writing an open textbook, but it works best if you want just a simple read through by the reader, in the manner of a traditional textbook. If though you want to make it more interactive, and open to comment, criticisms and substantive contributions from other people, then the current Pressbooks software is very limiting.

Pressbooks is a classic case of taking a new medium and merely transferring the format and structure of a previously existing medium. Although this is probably an essential and useful first step, what is really required is a complete re-design that fully exploits the characteristics or affordances of the new medium. For this to happen, though, a partnership between software engineers, potential authors and instructional designers is needed. However, there is a great opportunity here for creating truly innovative open source software for supporting open textbooks, if anyone has the time and resources to do this.

Authors such as myself also need to work out the difference (if any) between an open textbook and a learning management system. There are real difficulties in making everything in a course open, mainly because of hacking, spam and other external nuisances that can seriously disrupt a serious, engaged educational experience. The same applies to blogs and open textbooks. If the comment feature is too open it becomes overwhelmed with hacking and spam (I’m clearing about 50 bot-generated messages a day from my blog comment box – I don’t want to also have to spend this time keeping the comments on an open textbook under control.)

However, even accepting that an open textbook is not a substitute for an LMS, authors need to think carefully how the textbook can best be integrated or adopted within a course. Sample activities, suggestions for model answers, etc., can all be included. Above all, though, authors need to be clear when writing as to what will be done within the technological limitations of the textbook, what is best done outside the textbook, and how best to integrate these two elements.

I have to say I haven’t worked this out yet. It’s still a work in progress.

Over to you

As you can see, I am somewhat bumbling my way through the technology side of the writing, learning mainly through experience, although BCcampus has been more than helpful. I’d really like to hear though from other open textbook authors: is your experience similar or very different and if so why? Have you used different authoring software and how did that go?

Also, on the technology side, I’m still very open to other technology solutions, so long as they can be seamlessly integrated with Pressbooks. I have gone too far now to move to another software solution. But any suggestions welcome.

Is the classroom model appropriate for teaching in a digital age?

Listen with webReader
© University of Science and Arts Oklahoma

© University of Science and Arts Oklahoma

Chapter 5 of my open textbook, ‘Teaching in a Digital Age‘, is now published. In Chapter 5, I developed the concept of a learning environment.

I am now working on Chapter 6, ‘Models for Designing Teaching and Learning.’

What is to be covered in Chapter 6

This may change as I get into the writing but my plan at the moment is to cover the following topics:

  • What is a design model?
  • The classroom model
  • Classroom models in online learning
    • LMSs
    • lecture capture
  • ADDIE
  • Competency-based learning,
  • Communities of practice (inc. cMOOCs)
  • Flexible design models
  • PLEs
  • AI approaches.
  • Conclusion

In this post I introduce the concept of a design model and discuss the appropriateness of the classroom design model for a digital age. My next post, which follows almost immediately, does the same for the ADDIE model.

Purpose of the chapter

At the end of this chapter the reader should be able to:

  1. Describe key models or approaches to the design of teaching and learning
  2. Analyse each model in terms of its value for teaching in a digital age
  3. Decide which model or combination of models will fit best with their own teaching
  4. Use the model as a basis for designing their own teaching

What is a design model?

By a design model, I mean the organized steps taken to convert a desired learning environment into teaching and learning activities. Project management is a typical example of a design model, in that it presents a framework for taking a plan or goal and turning it into action. In project management, there are certain steps to be followed which are relatively independent of whatever project is being implemented.

However, there are many different kinds of approaches to design implementation besides project management. I intend to examine several of the most common design models that can be used in teaching, and in particular to examine them for their suitability for teaching in a digital age.

The classroom design model

Classroom old 2

Some design models are so embedded in tradition and convention that we are often like fish in water – we just accept that this is the environment in which we have to live and breath. The classroom model is a very good example of this. In a classroom based model, learners are organised in classes that meet on a regular basis at the same place at certain times of the day for a given length of time over a given period (a term or semester).

This is a design decision that was taken more than 150 years ago. It was embedded in the social, economic and political context of the 19th century. This context included:

  • the industrialization of society which provided ‘models’ for organizing both work and labour, such as factories and mass production
  • the movement of people from rural to urban occupations and communities, with increased density resulting in larger institutions
  • the move to mass education to meet the needs of industrial employers and an increasingly large and complex range of state-managed activities, such as government, health and education
  • voter enfranchisement and hence the need for a better educated voting public
  • over time, demand for more equality, resulting in universal access to education.

The large urban school, college or university, organized by age stratification, learners in groups, and regulated units of time was an excellent fit for such a society. In effect, we still have a predominantly factory model of educational design, which in large part remains our default design model even today.

However, over the span of 150 years, our society has slowly changed. Many of these factors or conditions no longer exist, while others persist, but often in a less dominant way than in the past. Thus we still have factories and large industries, but we also have many more small companies, greater social and geographical mobility, and above all a massive development of new technologies that allow both work and education to be organized in different ways. This is not to say that the classroom design model is inflexible. Teachers for many years have used a wide variety of teaching approaches within this overall model.

I don’t want to devote much space to the classroom design model, as we are all so familiar with it, and there is so much invested in the ‘default’ model that it is impractical to rip everything up and start with something completely different. Nevertheless, we have at least the seeds of change already showing. ‘Flipped’ classrooms where students get lectures on video and come to class for discussion and the re-design of large lecture classes are moves to modify the default model, while fully online programs and MOOCs are a manifestation of more radical change by offering education at any time and any place.

The real danger though is that we fail to grasp the opportunities that are now available to us, because we are so comfortable and familiar with the classroom design model. Even worse is trying to force the old default model on to new developments, when what is needed is a totally different approach if we are to meet the needs of a digital age. I give two examples below of forcing new technologies into the old classroom design model.

Old wine in new bottles: classroom-type online learning

When commercial movies were first produced, they were basically a transfer of previous music hall and vaudeville acts to the movie screen. Then along came D.W. Griffith’s ‘Birth of a Nation’, which transformed the design of movies, by introducing techniques that were unique to cinema at the time, such as panoramic long shots, panning shots, realistic battle scenes, and what are now known as special effects.

Learning management systems

Most learning management systems, such as Blackboard, Desire2Learn and Moodle, are in fact a replication of a classroom design model. They have weekly units or modules, the instructor selects and presents the material to all students in the class at the same time, a large class enrollment can be organized into smaller sections with their own instructors, there are opportunities for (online) discussion, students work through the materials at roughly the same pace, and assessment is by end-of-course tests or essays.

The main design differences are that the content is primarily text based rather than oral, the online discussion is asynchronous rather than synchronous, and the course content is available at any time from anywhere with an Internet connection. These are important differences, and skilled teachers and instructors can modify or adapt LMSs to meet different teaching or learning requirements (as they can in physical classrooms), but the basic organizing framework of the LMS remains the same as for a physical classroom.

Nevertheless, the LMS is still an advance over online designs that merely put lectures on the Internet or load up pdf copies of Powerpoint lecture notes, as is still the case unfortunately in many online programs. Good online design should take account of the special requirements of online learners, so the design needs to be different from that of a classroom model.

Lecture capture

This technology, which automatically records a classroom lecture, was originally designed to enhance the classroom model by making lectures available for repeat viewings online at any time for students regularly attending classes – in other words, a form of homework. Flipped classrooms are an attempt to exploit more fully this potential, but the biggest impact has been the use of lecture capture for ‘instructionist’ massive open online courses (MOOCs), such as those offered by Coursera, Udacity and edX. However, even this type of MOOC is really a basic classroom design model. The main differences are that the classroom is open to anyone (but then in principle so are many university lectures), and MOOCs are available to unlimited numbers at a distance. These are important differences again, but the design of the teaching – lectures delivered in chunks – has not changed markedly.

‘Instructionist’ MOOCs have resulted in some important design changes to the classroom model, such as using computer-marked assignments to test students or give feedback, and the use of peer review (both often used also in physical classroom design of course), but the predominant design model of instructionist MOOCs is that of an admittedly massive classroom.

The limitations of the classroom design model

Old wine can still be good wine, whether the bottle is new or not. What matters is whether classroom design meets the changing needs of a digital age. Just adding technology to the mix, or delivering the same design online, does not automatically result in meeting changing needs. It is important then to look at the design that makes the most of the educational affordances of new technologies, because unless the design changes significantly to take full advantage of the potential of the technology, the outcome is likely to be inferior to that of the physical classroom model which it is attempting to imitate.

The second danger of just adding new technology to the classroom design is that we may just be increasing cost, both in terms of technology and the time of instructors, without changing outcomes. Thus even if the new technology, such as lecture capture and computer-based multiple-choice questions organised in a MOOC, result in helping more students memorise better or learn more content, for example, this may not be sufficient to meet the higher level skills needed in a digital age.

Education is no exception to the phenomenon of new technologies being used at first merely to reproduce earlier design models before they find their unique potential. However, changes to the basic design model are needed if the demands of a digital age and the full potential of new technology are to be exploited in education.

Over to you

1. Do I manage to make clear what I mean by design ‘models’? If not, how can this be made clearer – or is the concept not helpful in the first place?

2. Do you agree that the classroom design model is a product of the 19th century and needs to changed for teaching in a digital age? Or is there still enough flexibility in the classroom model for our times?

3. To what extent do you feel you have to teach in a certain way because of the classroom model – or are you able to work flexibly within this model?

4. Do you agree that LMSs are basically a classroom model delivered online, or are they a unique design model in themselves. If so, what makes them unique?

What’s next?

My next post looks at the appropriateness of the ADDIE model for teaching in a digital age.

Adapting student assessment to the needs of a digital age

Listen with webReader

Assessment 2

The story so far

Chapter 5 of my open textbook, ‘Teaching in a Digital Age’ is about the design of teaching and learning, which I am currently writing and publishing as I go.

I started Chapter 5 by suggesting that instructors should think about design through the lens of constructing a comprehensive learning environment in which teaching and learning will take place. I have been working through the various components of a learning environment, focusing particularly on how the digital age affects the way we need to look at some of these components.

I briefly described some of the key components of an effective learning environment in a series of blog posts:

In this post, I examine the assessment of students as a key component, and how assessment methods need to be adapted to meet the needs of a digital age. This is the last component I’m discussing, but it will be followed by a final post that discusses the value of designing teaching and learning through the lens of a comprehensive learning environment.

Learner assessment

‘I was struck by the way assessment always came at the end, not only in the unit of work but also in teachers’ planning….Assessment was almost an afterthought…

Teachers…are being caught between competing purposes of …assessment and are often confused and frustrated by the difficulties that they experience as they try to reconcile the demands.’

Earle, 2003

Learner assessment in a digital age

Because assessment is a huge topic, it is important to be clear that the purpose of this section is (a) to look at one of the components that constitute an effective and comprehensive learning environment, and (b) briefly to examine the extent to which assessment is or should be changing in a digital age. Assessment will be a recurring theme in this book, so in this section the treatment is deliberately cursory.

Probably nothing drives the behaviour of students more than how they will be assessed. Not all students are instrumental in their learning, but given the competing pressures on students’ time in a digital age, most ‘successful’ learners focus on what will be examined and how they can most effectively (i.e. in as little time as possible) meet the assessment requirements. Therefore decisions about methods of assessment will in most contexts be fundamental to building an effective learning environment.

The purpose of assessment

There are many different reasons for assessing learners. It is important to be clear about the purpose of the assessment, because it is unlikely that one single assessment instrument will meet all assessment needs. Here are some reasons (you can probably think of many more):

  1. to improve and extend students’ learning
  2. to assess students’ knowledge and competence in terms of desired learning goals or outcomes
  3. to provide the teacher/instructor with feedback on the effectiveness of their teaching and how it might be improved
  4. to provide information for employers about what the student knows and/or can do
  5. to filter students for further study, jobs or professional advancement
  6. for institutional accountability and/or financial purposes.

I have deliberately ordered these in importance for creating an effective learning environment. In terms of the needs of a digital age, assessment needs to focus on both developing and assessing skills. This means that continuous or formative assessment will be as important as summative or ‘end-of-course’ assessment.

A question to be considered is whether there is a need for assessment of learning in the first place. There may be contexts, such as a community of practice, where learning is informal, and the learners themselves decide what they wish to learn, and whether they are satisfied with what they have learned. In other cases, learners may not want or need to be formally evaluated or graded, but do want or need feedback on how they are doing with their learning. ‘Do I really understand this?’ or ‘How am I doing compared to other learners?’

However, even in these contexts, some informal methods of assessment by experts, specialists or more experienced participants could help other participants extend their learning by providing feedback and indicating the level of competence or understanding that a participant has achieved or has yet to accomplish. Lastly, students themselves can extend their learning by participating in both self-assessment and peer assessment, preferably with guidance and monitoring from a more knowledgeable or skilled instructor.

Methods of assessment

The form the assessment takes, as well as the purpose, will be influenced by the instructors’ or examiners’ underlying epistemology: what they believe constitutes knowledge, and therefore how students need to demonstrate their knowledge. The form of assessment should also be influenced by the knowledge and skills that students need in a digital age, which means focusing as much on assessing skills as knowledge of content.

There is a wide range of possible assessment methods. I have selected just a few to illustrate how technology can change the way we assess learners in ways that are relevant to a digital age:

  • computer-based multiple-choice tests: good for testing ‘objective’ knowledge of facts, ideas, principles, laws, and quantitative procedures in mathematics, science and engineering etc., and are cost-effective for these purposes. This form of testing though tends to be limited  in assessing high-level intellectual skills, such as complex problem-solving, creativity, and evaluation, and therefore less likely to be useful for developing or assessing many of the skills needed in a digital age.
  • written essays or short answers: good for assessing comprehension and some of the more advanced intellectual skills, such as critical thinking, but are labour intensive, open to subjectivity, and are not good for assessing practical skills. Experiments are taking place with automated essay marking, using developments in artificial intelligence, but so far automated essay marking still struggles with reliably identifying valid semantic meaning (for a balanced and more detailed account of the current state of machine grading, see Mayfield, 2013Parachuri, 2013).
  • project work: either individual but more commonly group-based, project work encourages the development of authentic skills that require understanding of content, knowledge management, problem-solving, collaborative learning, evaluation, creativity and practical outcomes. Designing valid and practical project work needs a high level of skill and imagination from the instructor.
  • e-portfolios (an online compendium of student work): enables self-assessment through reflection, knowledge management, recording and evaluation of learning activities, such as teaching or nursing practice, and recording of an individual’s contribution to project work (as an example, see  the use of e-portfolios in Visual Arts and Built Environment at the University of Windsor.); usually self-managed by the learner but can be made available or adapted for formal assessment purposes or job interviews
  • simulations, educational games (usually online) and virtual worlds: facilitate the practice of skills, such as complex and real time decision-making, operation of (simulated or remote) complex equipment, the development of safety procedures and awareness, risk taking and assessment in a safe environment, and activities that require a combination of manual and cognitive skills (see the training of Canadian Border Service officers at Loyalist College, Ontario). Currently expensive to develop, but cost-effective with multiple use, where it replaces the use of extremely expensive equipment, where operational activities cannot be halted for training purposes, or  where available as open educational resources.
Virtual world border crossing, Loyalist College, Ontario

Virtual world border crossing, Loyalist College, Ontario

It can be seen that some of these assessment methods are both formative, in helping students to develop and increase their competence and knowledge, as well as summative, in assessing knowledge and skill levels at the end of a course or program.

In conclusion

Nothing is likely to drive student learning more than the method of assessment. At the same time, assessment methods are rapidly changing and are likely to continue to change. Assessment in terms of skills development needs to be both ongoing and continuous as well as summative. There is an increasing range of digitally based tools that can enrich the quality and range of student assessment. Therefore the choice of assessment methods, and their relevance to other components, are vital elements of any effective learning environment.

Over to you

Your views, comments and criticisms are always welcome. In particular:

  • are there other methods of assessment relevant to a digital age that I should have included?
  • there is still a heavy reliance on computer-based multiple-choice tests in much teaching, mainly for cost reasons. However, although there are exceptions, in general these really don’t assess the high level conceptual skills needed in a digital age. Are there other methods that are equally as economical, particularly in terms of instructor time, that are more suitable for assessment in a digital age? For instance, do you think automated essay grading is a viable alternative?
  • would it be helpful to think about assessment right at the start of course planning, rather than at the end? Is this feasible?

Or any other comments on assessment as a critical component of a learning environment, please!

Next up

Why thinking in terms of a comprehensive learning environment is necessary but not sufficient when designing a course or program.

 

References

Earle, L. (2003) Assessment as Learning Thousand Oaks CA: Corwin Press

Mayfield, E. (2013) Six ways the edX Announcement Gets Automated Essay Grading Wrong, e-Literate, April 8

Parachuri, V. (2013) On the automated scoring of essays and the lessons learned along the way, vicparachuri.com,  July 31