November 22, 2014

Adapting student assessment to the needs of a digital age

Listen with webReader

Assessment 2

The story so far

Chapter 5 of my open textbook, ‘Teaching in a Digital Age’ is about the design of teaching and learning, which I am currently writing and publishing as I go.

I started Chapter 5 by suggesting that instructors should think about design through the lens of constructing a comprehensive learning environment in which teaching and learning will take place. I have been working through the various components of a learning environment, focusing particularly on how the digital age affects the way we need to look at some of these components.

I briefly described some of the key components of an effective learning environment in a series of blog posts:

In this post, I examine the assessment of students as a key component, and how assessment methods need to be adapted to meet the needs of a digital age. This is the last component I’m discussing, but it will be followed by a final post that discusses the value of designing teaching and learning through the lens of a comprehensive learning environment.

Learner assessment

‘I was struck by the way assessment always came at the end, not only in the unit of work but also in teachers’ planning….Assessment was almost an afterthought…

Teachers…are being caught between competing purposes of …assessment and are often confused and frustrated by the difficulties that they experience as they try to reconcile the demands.’

Earle, 2003

Learner assessment in a digital age

Because assessment is a huge topic, it is important to be clear that the purpose of this section is (a) to look at one of the components that constitute an effective and comprehensive learning environment, and (b) briefly to examine the extent to which assessment is or should be changing in a digital age. Assessment will be a recurring theme in this book, so in this section the treatment is deliberately cursory.

Probably nothing drives the behaviour of students more than how they will be assessed. Not all students are instrumental in their learning, but given the competing pressures on students’ time in a digital age, most ‘successful’ learners focus on what will be examined and how they can most effectively (i.e. in as little time as possible) meet the assessment requirements. Therefore decisions about methods of assessment will in most contexts be fundamental to building an effective learning environment.

The purpose of assessment

There are many different reasons for assessing learners. It is important to be clear about the purpose of the assessment, because it is unlikely that one single assessment instrument will meet all assessment needs. Here are some reasons (you can probably think of many more):

  1. to improve and extend students’ learning
  2. to assess students’ knowledge and competence in terms of desired learning goals or outcomes
  3. to provide the teacher/instructor with feedback on the effectiveness of their teaching and how it might be improved
  4. to provide information for employers about what the student knows and/or can do
  5. to filter students for further study, jobs or professional advancement
  6. for institutional accountability and/or financial purposes.

I have deliberately ordered these in importance for creating an effective learning environment. In terms of the needs of a digital age, assessment needs to focus on both developing and assessing skills. This means that continuous or formative assessment will be as important as summative or ‘end-of-course’ assessment.

A question to be considered is whether there is a need for assessment of learning in the first place. There may be contexts, such as a community of practice, where learning is informal, and the learners themselves decide what they wish to learn, and whether they are satisfied with what they have learned. In other cases, learners may not want or need to be formally evaluated or graded, but do want or need feedback on how they are doing with their learning. ‘Do I really understand this?’ or ‘How am I doing compared to other learners?’

However, even in these contexts, some informal methods of assessment by experts, specialists or more experienced participants could help other participants extend their learning by providing feedback and indicating the level of competence or understanding that a participant has achieved or has yet to accomplish. Lastly, students themselves can extend their learning by participating in both self-assessment and peer assessment, preferably with guidance and monitoring from a more knowledgeable or skilled instructor.

Methods of assessment

The form the assessment takes, as well as the purpose, will be influenced by the instructors’ or examiners’ underlying epistemology: what they believe constitutes knowledge, and therefore how students need to demonstrate their knowledge. The form of assessment should also be influenced by the knowledge and skills that students need in a digital age, which means focusing as much on assessing skills as knowledge of content.

There is a wide range of possible assessment methods. I have selected just a few to illustrate how technology can change the way we assess learners in ways that are relevant to a digital age:

  • computer-based multiple-choice tests: good for testing ‘objective’ knowledge of facts, ideas, principles, laws, and quantitative procedures in mathematics, science and engineering etc., and are cost-effective for these purposes. This form of testing though tends to be limited  in assessing high-level intellectual skills, such as complex problem-solving, creativity, and evaluation, and therefore less likely to be useful for developing or assessing many of the skills needed in a digital age.
  • written essays or short answers: good for assessing comprehension and some of the more advanced intellectual skills, such as critical thinking, but are labour intensive, open to subjectivity, and are not good for assessing practical skills. Experiments are taking place with automated essay marking, using developments in artificial intelligence, but so far automated essay marking still struggles with reliably identifying valid semantic meaning (for a balanced and more detailed account of the current state of machine grading, see Mayfield, 2013Parachuri, 2013).
  • project work: either individual but more commonly group-based, project work encourages the development of authentic skills that require understanding of content, knowledge management, problem-solving, collaborative learning, evaluation, creativity and practical outcomes. Designing valid and practical project work needs a high level of skill and imagination from the instructor.
  • e-portfolios (an online compendium of student work): enables self-assessment through reflection, knowledge management, recording and evaluation of learning activities, such as teaching or nursing practice, and recording of an individual’s contribution to project work (as an example, see  the use of e-portfolios in Visual Arts and Built Environment at the University of Windsor.); usually self-managed by the learner but can be made available or adapted for formal assessment purposes or job interviews
  • simulations, educational games (usually online) and virtual worlds: facilitate the practice of skills, such as complex and real time decision-making, operation of (simulated or remote) complex equipment, the development of safety procedures and awareness, risk taking and assessment in a safe environment, and activities that require a combination of manual and cognitive skills (see the training of Canadian Border Service officers at Loyalist College, Ontario). Currently expensive to develop, but cost-effective with multiple use, where it replaces the use of extremely expensive equipment, where operational activities cannot be halted for training purposes, or  where available as open educational resources.
Virtual world border crossing, Loyalist College, Ontario

Virtual world border crossing, Loyalist College, Ontario

It can be seen that some of these assessment methods are both formative, in helping students to develop and increase their competence and knowledge, as well as summative, in assessing knowledge and skill levels at the end of a course or program.

In conclusion

Nothing is likely to drive student learning more than the method of assessment. At the same time, assessment methods are rapidly changing and are likely to continue to change. Assessment in terms of skills development needs to be both ongoing and continuous as well as summative. There is an increasing range of digitally based tools that can enrich the quality and range of student assessment. Therefore the choice of assessment methods, and their relevance to other components, are vital elements of any effective learning environment.

Over to you

Your views, comments and criticisms are always welcome. In particular:

  • are there other methods of assessment relevant to a digital age that I should have included?
  • there is still a heavy reliance on computer-based multiple-choice tests in much teaching, mainly for cost reasons. However, although there are exceptions, in general these really don’t assess the high level conceptual skills needed in a digital age. Are there other methods that are equally as economical, particularly in terms of instructor time, that are more suitable for assessment in a digital age? For instance, do you think automated essay grading is a viable alternative?
  • would it be helpful to think about assessment right at the start of course planning, rather than at the end? Is this feasible?

Or any other comments on assessment as a critical component of a learning environment, please!

Next up

Why thinking in terms of a comprehensive learning environment is necessary but not sufficient when designing a course or program.

 

References

Earle, L. (2003) Assessment as Learning Thousand Oaks CA: Corwin Press

Mayfield, E. (2013) Six ways the edX Announcement Gets Automated Essay Grading Wrong, e-Literate, April 8

Parachuri, V. (2013) On the automated scoring of essays and the lessons learned along the way, vicparachuri.com,  July 31

 

Research and development in online learning from the Open University of Catalonia

Listen with webReader

Media TIC, home of the eLearn Centre

Two weeks ago I visited two open universities (the UK OU and the Open University of Catalonia) and a research lab of the Institute of Education at the University of London. Full reports of the visits will be appearing later this month on the Ontario Online Learning Portal for Faculty and Instructors managed by Contact North. These reports are looked at from the perspective of key ‘game-changers’ in online learning, and provide an overall picture of each institution.

However, I want to use my blog to discuss in more detail the research into online learning that is being conducted in these institutions, because as a result of these visits I want to question why here in Canada we are so disorganized and frankly ineffective in the way we conduct research in this area, despite having several world leaders in online learning research and development.

First though I will provide a series of posts on the research and development being done at the three European institutions we visited. The first post is on research at the Universitat Oberta de Catalunya (UOC), where there are two research and development units as well as a program of innovation specifically on online learning.

the eLearn Centre.

The director is Albert Sangrà, a former colleague of mineThis unit has 10 full time researchers and 134 researchers across the world (including some from Canada) affiliated or collaborating on research with staff from the eLearn Centre.

The time factor

The eLearn Centre decided to focus on a four year program of research on the time factor in online learning. The four year period ends this month. This study covers topics as diverse as learning rhythms, the timing of curricula and courses, student time management, and the effect of timing on feedback and learning. The research team leader is Elena Barberà.

Research students doing dissertations as part of their graduate studies in the eLearn Centre, and affiliated researchers, have been asked to include a least a question on the time factor when collecting data and analysing their results, whatever the topic of their thesis or dissertation. As a result the centre has been able to produce a six monthly journal on the time factor in online learning.

There are now five issues of this journal published to date, with 28 different papers published, covering the time factor in assessment, collaborative learning, time management in networked learning, and the time factor in online teaching and learning in maths and physics.

Open educational resources

The eLearn Centre is a partner with several other European universities in a European Commission project called OERtest, whose objectives are:

  • creation of a single portal for accessing Euro-centric OER content
  • development of quality standards, assessment guidelines, financial models, curricular provisions and any other administrative requirements necessary to allow for HEIs within the EU to assess learning received exclusively through OER
  • assessment of the feasibility for EU HEIs to offer assessment services for OER
  • establishment of a European network to promote and follow the development of OER and Open Educational Practices within the EHEA.

eLearn centre staff are also engaged in another European Commission project, OporTunidad. The project intends to foster the adoption and pilot of open educational practices, and open educational resources), at an institutional level, in Latin American countries. The focus here is on institutional strategies that promote the adoption and use of OEPs and OERs. Contact at UOC: Lourdes Guàrdia

e-portfolios

Another focus of research is on e-portfolios. The centre has taken a lead role in developing a Spanish national community of practice on the use of e-portfolios in post-secondary education, with 14 institutional members, with a focus particularly on the use of e-portfolios for assessment. Staff from the eLearn centre are also particpating in another project funded by the European Commission, Europortfolio. The aim is to create a Learning Community Portal as a space to publish, share and review data and resources on ePortfolio practices and technologies across Europe. Contact at UOC: Lourdes Guàrdia

Dissemination

eLearn Centre staff disseminate their research and experience through establishing a community of practice for UOC faculty for training in how best to use OERs and e-portfolios, as well as drawing on the research for more formal teaching such as in UOC’s Masters in e-Learning.

The eLearn centre also worked in collaboration with the New Media Consortium to produce the Iberoamerican edition of the Horizon Report 2010, which specifically looked at the Spanish/Latin American context, and has an invited visiting scholars program, and a program for inviting institutions to visit.

In addition to its research, the eLearn Centre also provides training in e-learning through its Doctorate program, and its Masters, Diplomas and Certificates in Education and ICTs.

I have touched on only part of the work of the eLearn Centre. There are 10 other research groups associated with the eLearn Centre. More details can be found at: http://www.uoc.edu/portal/en/elearncenter/index.html

A 'xarcuteria' on Carrer Casanova, Barcelona

The Office of Learning Technologies

Its mission is ‘to create the learning environments of the 21st Century for the new digital generations and global citizens.’ It has a staff of 42, and its director is Magì Almirall. This is an educational technology development group. This department develops a wide variety of tools and applications for use in the university. In 2011 it was working on a total of 38 projects.

A major focus at the moment is the development of ‘My mobile UOC’, that enables students to access their learning on any kind of devices, websites and other environments such as SmartTV or Chrome Operating System. Several of the projects focus on helping students with disabilities, by making the online environment more accessible.

The Office has developed a number of social media applications and tools, such as microblogging tools and small group online videoconferencing facilities, as well as augemented reality tools for creating virtual worlds.

All these tools are integrated or interoperable with the university’s in-house developed Virtual Campus, an open source, combined learning management and administrative system. Once the tools developed by the office become adopted and operational, the responsibility for maintaining them passes to the Learning Services division. However, the Office is also responsible for the overall design of the university portal and the community services that are run through the portal.

The Office of Learning Technologies reports to the Vice Rector, Technology, who manages a fund of around 100,000 euros a year for innovative projects that are bid for internally through an RFP process. The theme this year has been mobile learning, including the development of apps for learning.

Summary

The Open University of Catalonia is a fully online university with more than 60,000 students and an annual operating budget of 100 million euros ($125 million). This has enabled it to set up these R&D units at a sufficient scale of operation that they can take on substantial projects that will have direct impact on the operation of the university, both pedagogically and technically. Given that both these units are relatively new, their influence on the external world of online learning is likely to grow, despite language and cultural differences.

A Modernista building on the Rambla Catalunya

 

 

Innovation in teaching in Ontario universities

Listen with webReader

 

Council of Ontario Universities (2012) Beyond the sage on the stage: innovative and effective teaching and learning in Ontario universities Toronto ON: COU

This document provides an overview and analysis of innovative teaching practices in Ontario’s 24 universities:

There are many examples of innovative and effective teaching and learning strategies at Ontario universities, some of which will be shared here. The examples set out in this document reflect both practices that are well-established in many universities, and those that are evolving. 

With regard to online learning and the use of learning technologies, the document lists the following examples (this is a selection of what I found the most interesting – for example, I don’t find the use of clickers innovative):

  • Video recording: 

Carleton University: Robert Burk’s General Chemistry course (700 students): lectures, tutorials and other course materials are broadcast via cable television, webcast, video on demand, iTunes and a course website, combined with personal e-mails to every student. The materials are the most downloaded items from Carleton University; some have been downloaded almost 250,000 times in two years. A full third of his students never set foot in his lecture hall, yet their grades are identical to the two-thirds who are studying on campus. For an example, see: Making Nylon

  • Hybrid learning: 

Lakehead University: Dr. Glenna Knutson: the Masters in Public Health, designed initially to serve the needs of public health professionals across northwestern Ontario, uses WebCT and media streaming to ensure that distance students, who make up three-quarters of the class, can participate fully. In addition to taking part in large and small group discussions during class, they can use the technology platform to work with classmates outside of class time, preparing projects and presentations. To further accommodate the professional and family commitments of students, the program provides the option of completing it in six terms, or even 12, to make it more flexible.

  • Digital entrepreneurs: 

Ryerson University: The Digital Media Zone is a business incubator that supports digital entrepreneurs with business knowledge, resources and, above all, space to work and collaborate. It was the brainchild of President Sheldon Levy, who saw the need for universities to go beyond helping students find jobs. DMZ also focuses on helping students create the jobs and companies of the future. Since its launch in 2010, it has grown to accommodate some 200 innovators, spawning more than 40 companies and creating over 400 jobs in the process

University of Waterloo: the  VeloCity Mobile and Media incubator residence is the world’s first student residence designed to enable budding entrepreneurs to work with like-minded colleagues on mobile communications and digital media. It is a “dorm-cubator” for top students who want to turn their bright ideas in web, mobile and digital media applications into successful businesses. The value of companies created by VeloCity alumni is estimated to be about $50 million based on initial feedback from over 200 alumni who have lived in VeloCity. Participation in this program builds a supportive community that helps students succeed. Outcomes measured are not grades, but rather the success that students have both personally and professionally, by engaging in the business world outside of the institution. VeloCity incorporates peer mentorship and connects students  to the world of global start-up hubs (Waterloo, Toronto, Montreal, Ottawa, Boston, New York, San Francisco). Students learn how to manage risks, focus their skills and decide if building their own company is something they want to do after they graduate or even while they are still at the undergraduate level.

  • Universal Instructional Design

Trent University: The Transcribe Your Class project is an example of how the benefits of Universal Instructional Design modifications to a course can extend to all the students in a course, beyond students who have a learning disability. Through this project students with disabilities attending post-secondary education and National Disability Organizations use advanced Speech-Recognition Technology to improve access to information. Lectures are first recorded as webcasts through a software program, Panopto 7, and then transcribed. The transcriptions are integrated into a multimedia platform, which includes audio, video and presentation slides. The transcribed text is also searchable within the Panopto platform. At present, six first-year courses are included in this project. Prior to the implementation of the Transcribe Your Class project, students who required an accommodation for speech-to-text transcriptions worked with the Disability Services Office to have lectures recorded using digital audio recorders, and then paid a commercial firm to have them transcribed. The Transcribe Your Class project means that the instructor can automatically record lectures with a touch of a button. The recordings are uploaded immediately after the lecture and sent to IBM for speech-to-text recognition. The transcribed lectures are available to students within 48-96 hours of the original recording. This  is a significant improvement over the typical five-day turnaround time for edited transcripts through commercial services. The transcripts are made available to all students enrolled in the course.

  • E-portfolios

University of Guelph: E-portfolio use is incorporated throughout the Bachelor of Arts and Science Program. The basic function of the e-portfolio is to serve as a repository where students can compile their course work, writing and other material, including material from internships and other types of placements. E-portfolios enable students to engage in a process of reflection about the knowledge and skills they have acquired in their program of studies, and provide students with a useful tool for making connections about what they are learning.Both faculty and students report that they get to know each other better through“About Me” pages that are constructed in e-portfolio. Senior students may develop personalized e-portfolios to showcase their education and skills to prospective employers, and for applications to post-graduate programs.

Wilfred Laurier University: Kimberley Barber of the Faculty of Music has initiated an e-portfolio for her first-year voice performance students. Throughout the first term, students complete weekly e-portfolio presentations, including logs of their practice sessions and reflections on that practice, and their performances to help them evaluate their strengths and areas for improvement. They are also encouraged to upload digitized files of their performances to the e-portfolio system so that, over the course of their four-year program, they will be able to review their work and see their own progress. Student self-evaluation and critique are essential in the development of musical skills for both performance and education; early results have shown it to be a very useful pedagogy. E-portfolios also enable students to assess their entire university education holistically. This system is an efficient method for both compiling work and exchanging assignments and information between professors and students. There is no need for the exchange of paper documents, and students can receive feedback quickly from their professor and/or peers.

For another 40 examples of innovative teaching in Ontario universities and colleges, see: Pockets of Innovation from Contact North

Comment

First, kudos to COU for showing that there is much more going on in Ontario universities than just boring lectures. Having examples of the ways campus-based institutions are integrating technology is always very useful.

Second, what does ‘innovative teaching’ really mean? Certainly, for those instructors who have developed these approaches, it will certainly be innovative. However, one man’s meat is another man’s poison. Don’t get me going on clickers, for example. They are about as innovative as a caveman waving his club. For many readers of this blog, the reaction to some of these examples is likely to be a shrug of the shoulders; for some, it may reinforce your own ideas of where your teaching should go; for others it will, I hope, provide a spark that will lead to your own innovation in teaching.

Third, there were many other examples in this document of innovative teaching that did not involve any technology. They were about different pedagogical approaches (e.g. inquiry-based learning, applied and practical learning, and new ways of providing professional development.) This reinforces my view that just using technology is not innovative, even if the technology is new. It has to do something different and better, in terms of teaching and learning.

Just a couple of negative points. First, where were the formal evaluations of these projects? This is more an institutional responsibility. Innovations in teaching should be independently evaluated, and if successful, efforts should be made to spread the innovation beyond the innovator. Second, what is the institution’s overall strategy for supporting innovation? The COU says, as a body representing universities, that it supports innovation in teaching on principle, but moving beyond individual pockets of innovation to a culture of innovation across an institution needs more than a pat on the head as a strategy. Developing a strategy for innovation is a responsibility of senior academic management.

Nevertheless, it is good to see universities not only responding to the need for innovation in teaching and learning, but also letting everyone know what they are doing. We can all learn something from this document.


 

E-portfolios, Sakai, and ‘new paradigms’

Listen with webReader

Batson, T. (2012) The Wait is Over: The LMS and the ePortfolio Merge to Serve a Culture of Learning Campus Technology, May 16

In this article, Batson argues that Sakai’s OAE (Open Academic Environment – formerly Sakai 3) reflects a new paradigm of online learning, moving from an instructor-led to a learner-led environment based around its e-portfolio capability.

However, surely this will depend on how instructors choose to use it? A true learner-focused environment is likely to include more than one proprietary set of tools. See ‘Designing online learning for the 21st century‘ for an alternative view.

5 video case studies of e-portfolio implementation + an implementation toolkit

Listen with webReader

 

JISC has created a site with videos from five UK universities and colleges that document their experience in implementing e-portfolios across a relatively wide range of subject areas.

In addition, this site also provides an e-portfolio implementation tool kit, based on the experiences of its case study contributors: 12 UK, 4 Australian and 3 New Zealand partner institutions together with one professional organisation. The kit contains the following:

The e-portfolio implementation toolkit

Background

Guidance

Case Studies

Additional resources

This complements JISC’s 2008 publication, Effective Practice with e-Portfolios.

Thanks to Richard Elliott’s e-Learning Watch for directing me to this.