November 28, 2014

Adapting student assessment to the needs of a digital age

Listen with webReader

Assessment 2

The story so far

Chapter 5 of my open textbook, ‘Teaching in a Digital Age’ is about the design of teaching and learning, which I am currently writing and publishing as I go.

I started Chapter 5 by suggesting that instructors should think about design through the lens of constructing a comprehensive learning environment in which teaching and learning will take place. I have been working through the various components of a learning environment, focusing particularly on how the digital age affects the way we need to look at some of these components.

I briefly described some of the key components of an effective learning environment in a series of blog posts:

In this post, I examine the assessment of students as a key component, and how assessment methods need to be adapted to meet the needs of a digital age. This is the last component I’m discussing, but it will be followed by a final post that discusses the value of designing teaching and learning through the lens of a comprehensive learning environment.

Learner assessment

‘I was struck by the way assessment always came at the end, not only in the unit of work but also in teachers’ planning….Assessment was almost an afterthought…

Teachers…are being caught between competing purposes of …assessment and are often confused and frustrated by the difficulties that they experience as they try to reconcile the demands.’

Earle, 2003

Learner assessment in a digital age

Because assessment is a huge topic, it is important to be clear that the purpose of this section is (a) to look at one of the components that constitute an effective and comprehensive learning environment, and (b) briefly to examine the extent to which assessment is or should be changing in a digital age. Assessment will be a recurring theme in this book, so in this section the treatment is deliberately cursory.

Probably nothing drives the behaviour of students more than how they will be assessed. Not all students are instrumental in their learning, but given the competing pressures on students’ time in a digital age, most ‘successful’ learners focus on what will be examined and how they can most effectively (i.e. in as little time as possible) meet the assessment requirements. Therefore decisions about methods of assessment will in most contexts be fundamental to building an effective learning environment.

The purpose of assessment

There are many different reasons for assessing learners. It is important to be clear about the purpose of the assessment, because it is unlikely that one single assessment instrument will meet all assessment needs. Here are some reasons (you can probably think of many more):

  1. to improve and extend students’ learning
  2. to assess students’ knowledge and competence in terms of desired learning goals or outcomes
  3. to provide the teacher/instructor with feedback on the effectiveness of their teaching and how it might be improved
  4. to provide information for employers about what the student knows and/or can do
  5. to filter students for further study, jobs or professional advancement
  6. for institutional accountability and/or financial purposes.

I have deliberately ordered these in importance for creating an effective learning environment. In terms of the needs of a digital age, assessment needs to focus on both developing and assessing skills. This means that continuous or formative assessment will be as important as summative or ‘end-of-course’ assessment.

A question to be considered is whether there is a need for assessment of learning in the first place. There may be contexts, such as a community of practice, where learning is informal, and the learners themselves decide what they wish to learn, and whether they are satisfied with what they have learned. In other cases, learners may not want or need to be formally evaluated or graded, but do want or need feedback on how they are doing with their learning. ‘Do I really understand this?’ or ‘How am I doing compared to other learners?’

However, even in these contexts, some informal methods of assessment by experts, specialists or more experienced participants could help other participants extend their learning by providing feedback and indicating the level of competence or understanding that a participant has achieved or has yet to accomplish. Lastly, students themselves can extend their learning by participating in both self-assessment and peer assessment, preferably with guidance and monitoring from a more knowledgeable or skilled instructor.

Methods of assessment

The form the assessment takes, as well as the purpose, will be influenced by the instructors’ or examiners’ underlying epistemology: what they believe constitutes knowledge, and therefore how students need to demonstrate their knowledge. The form of assessment should also be influenced by the knowledge and skills that students need in a digital age, which means focusing as much on assessing skills as knowledge of content.

There is a wide range of possible assessment methods. I have selected just a few to illustrate how technology can change the way we assess learners in ways that are relevant to a digital age:

  • computer-based multiple-choice tests: good for testing ‘objective’ knowledge of facts, ideas, principles, laws, and quantitative procedures in mathematics, science and engineering etc., and are cost-effective for these purposes. This form of testing though tends to be limited  in assessing high-level intellectual skills, such as complex problem-solving, creativity, and evaluation, and therefore less likely to be useful for developing or assessing many of the skills needed in a digital age.
  • written essays or short answers: good for assessing comprehension and some of the more advanced intellectual skills, such as critical thinking, but are labour intensive, open to subjectivity, and are not good for assessing practical skills. Experiments are taking place with automated essay marking, using developments in artificial intelligence, but so far automated essay marking still struggles with reliably identifying valid semantic meaning (for a balanced and more detailed account of the current state of machine grading, see Mayfield, 2013Parachuri, 2013).
  • project work: either individual but more commonly group-based, project work encourages the development of authentic skills that require understanding of content, knowledge management, problem-solving, collaborative learning, evaluation, creativity and practical outcomes. Designing valid and practical project work needs a high level of skill and imagination from the instructor.
  • e-portfolios (an online compendium of student work): enables self-assessment through reflection, knowledge management, recording and evaluation of learning activities, such as teaching or nursing practice, and recording of an individual’s contribution to project work (as an example, see  the use of e-portfolios in Visual Arts and Built Environment at the University of Windsor.); usually self-managed by the learner but can be made available or adapted for formal assessment purposes or job interviews
  • simulations, educational games (usually online) and virtual worlds: facilitate the practice of skills, such as complex and real time decision-making, operation of (simulated or remote) complex equipment, the development of safety procedures and awareness, risk taking and assessment in a safe environment, and activities that require a combination of manual and cognitive skills (see the training of Canadian Border Service officers at Loyalist College, Ontario). Currently expensive to develop, but cost-effective with multiple use, where it replaces the use of extremely expensive equipment, where operational activities cannot be halted for training purposes, or  where available as open educational resources.
Virtual world border crossing, Loyalist College, Ontario

Virtual world border crossing, Loyalist College, Ontario

It can be seen that some of these assessment methods are both formative, in helping students to develop and increase their competence and knowledge, as well as summative, in assessing knowledge and skill levels at the end of a course or program.

In conclusion

Nothing is likely to drive student learning more than the method of assessment. At the same time, assessment methods are rapidly changing and are likely to continue to change. Assessment in terms of skills development needs to be both ongoing and continuous as well as summative. There is an increasing range of digitally based tools that can enrich the quality and range of student assessment. Therefore the choice of assessment methods, and their relevance to other components, are vital elements of any effective learning environment.

Over to you

Your views, comments and criticisms are always welcome. In particular:

  • are there other methods of assessment relevant to a digital age that I should have included?
  • there is still a heavy reliance on computer-based multiple-choice tests in much teaching, mainly for cost reasons. However, although there are exceptions, in general these really don’t assess the high level conceptual skills needed in a digital age. Are there other methods that are equally as economical, particularly in terms of instructor time, that are more suitable for assessment in a digital age? For instance, do you think automated essay grading is a viable alternative?
  • would it be helpful to think about assessment right at the start of course planning, rather than at the end? Is this feasible?

Or any other comments on assessment as a critical component of a learning environment, please!

Next up

Why thinking in terms of a comprehensive learning environment is necessary but not sufficient when designing a course or program.

 

References

Earle, L. (2003) Assessment as Learning Thousand Oaks CA: Corwin Press

Mayfield, E. (2013) Six ways the edX Announcement Gets Automated Essay Grading Wrong, e-Literate, April 8

Parachuri, V. (2013) On the automated scoring of essays and the lessons learned along the way, vicparachuri.com,  July 31

 

Teaching practice in a virtual world

Listen with webReader
UCF demo of software

UCF demo of software

Ortega, S. (2014) UCF teaching technology spreads to 33 universities Central Florida Future, January 6

I haven’t heard a lot in the last year about virtual worlds, so I was interested to see that faculty at the University of Central Florida have developed a virtual world/simulator for student teachers. The aim is to ‘simulate teacher practice aid, teacher education and research and development…. TLE TeachLivE™ is primarily a virtual classroom software designed to help teachers …train and prepare for an actual classroom setting. Automated student atavars react to student input, thus avoiding real children being negatively affected by inappropriate student activities.

It has been found specially useful for training teachers to deal with special needs children, such as children with autism. This doesn’t replace activities with real children, but is combined with other teaching practice. ‘I wanted somewhere that teachers could practice specifically targeted skills without having to put students, especially those with disabilities, at risk of feeling upset or bad and putting more responsibility on the teacher,” said LisaDieker, an education professor at UCF.

TLE TeachLivE™ is  being used in 33 universities and has already helped train over 10,000 teachers and as a consequence has impacted on over half a million school children.

Examples of virtual worlds, simulations and mobile apps from Ontario

Listen with webReader

Carleton Virtual

This is the second in a series of guest blogs on innovative developments in online learning in Ontario post-secondary institutions. (The first was examples of hybrid learning.)

In this post, Judith Tobin of Contact North| Contact Nord focuses on examples of virtual worlds and simulations, and mobile apps. Here is her guest post:

Introduction

In my previous guest blog, I highlighted examples of online and hybrid learning innovations in Ontario post-secondary institutions that were serving as vehicles for bringing new ideas into the classroom, such as restructuring class time, collaborative learning, and changes in the teacher/student relationships. This time, I have focused on applications of leading edge technologies, including virtual environments, simulations, and mobile apps. These technologies are just beginning to have a role in learning and considerable research and experimentation are taking place to determine their optimal contributions. The innovations below explore student learning and responses, while working to make the technologies easy to use and flexible.

Virtual Worlds and Simulations

The Virtual World Design Centre at Loyalist College in Belleville has been working with virtual environments for a number of years, using the open source software Second Life (http://secondlife.com) since 2006.  They have recently adopted the Unity 3D authoring tool as they find it more adaptable for educational purposes. 

 Virtual worlds are successful in education because students identify with the characters and the situations portrayed and so become active participants in the events on screen. The learning from these experiences carries over into real life applications.  In an award-winning and educationally successful project, the staff in the Virtual World Design Centre created a virtual border crossing at which students’ avatars take on the roles of border crossing guards, interviewing travellers who present challenges of documentation, prohibitions, smuggling, and difficult communication.  The virtual traveler interviews take place in class and each encounter is then analyzed by the entire group so that best practices are identified.  Applications for completely online learning are being investigated.


Virtual environment of a border crossing

The students at Loyalist found the virtual experience provided them with more than enhanced content learning; they also developed confidence, observational skills, and the capacity to respond to developing situations.

 Other virtual environments created at Loyalist included a virtual hospital tour for secondary school students and a factory simulation for the repair of machinery for food processing. Experience in these virtual worlds prepares students for more effective and informed exposure to the real environments.

Carleton Virtual is an online virtual environment resembling the physical setting of Carleton University in Ottawa that was constructed to explore how virtual environment can be used to enhance learning.  An English-as-a-second-language teacher used the virtual meeting space so that her students could practice language usage and collaboration skills in a risk-free environment. They used the virtual classrooms, meeting rooms, and other spaces, with many students participating more actively in the virtual spaces than in a face-to-face classroom.  A virtual archeological site allowed students to better understand the archeological processes that create knowledge, when a visit to a functioning site was not possible.

The virtual archeological site

At the Sheridan Institute of Technology and Advanced Learning in Oakville, students in the Veterinary Technician Program could no longer have access to live animals and cadavers for both ethical and health reasons. In response to this need, the Network for Innovation and Leadership in Education built a 3D dog skeleton with images that can be manipulated and disassembled. The skeleton was then built into a web-based learning tool as a complete package for online student learning, including mobile applications.  The simulation is being tested for educational effectiveness.

Mobile learning

In Fall 2012, all full-time, first-year students enrolled in the four-year Bachelor of Business Administration Program at Nipissing University in North Bay will receive iPads. Leading up to this innovation, pilot projects are being undertaken to explore applications for teaching and learning.  For example, connectivity was set up so a professor could use an iPad while moving about the room and illustrating and annotating normally static PDF documents, then saving the revised slides for class distribution. Meanwhile, the students would use their iPads to add their own notes to the same slides. The first year of the iPad usage will be dedicated to introducing students to software for document preparation and research; more advanced learning applications will be developed over time to emphasize the collaborative and creative possibilities of the iPad.

Nipissing is also exploring the development of apps for mobile learning but, for ease of access and flexibility of revision, these would be web applications hosted on web servers that would look and function like apps. 

Using mobile apps to encourage language learning and practice outside the classroom has been the focus of recent work at George Brown College. Extensive research is being done on student ownership of mobile devices and data plans, preferences for activities in mobile learning, and an instructional design framework to encourage active participation and extensive language usage. From this, a set of design principles for effective mobile learning was created. The Mobile Learning Specialist at George Brown would like to collaborate with colleagues in other institutions on mobile learning developments.

Research and evaluation

Research is an essential component of all these innovations as post-secondary institutions ponder and test how these technological and software advances can best be used to serve learners.  They are often expensive and require technological and pedagogical sophistication to develop and implement. The educators want to be sure they are using them as tools for effective learning.

Thank you, Judith, and if anyone else would like to do a guest post of an innovation in online learning at their institution, please contact me at tony.bates@ubc.ca

Further reading

For more details on each application go to the following links:

Border Simulation – Student Learning in a Virtual World: Loyalist College

Simulations for Learning: Loyalist College

Carleton Virtual: Carleton University

A Simulation-Based Learning Tool for Students in the Veterinary Technician Diploma Program: Sheridan Institute of Technology and Advanced Learning

Going mobile: Nipissing University

Mobile-assisted language learning: George Brown College

For more examples of innovation in online learning in Ontario, go to: Pockets of Innovation

 

 

JET&S: Special journal issue on technology supported cognition and exploratory learning

Listen with webReader

The Journal of Educational Technology and Society, Vol. 15, No.1, has a special edition on technology supported cognition and exploratory learning.

From the editorial:

The IADIS CELDA 2010 conference aims to address the main issues concerned with evolving learning processes and supporting pedagogies and applications in the digital age. There have been advances in both cognitive psychology and computing that have affected the educational arena. The convergence of these two disciplines is increasing at a fast pace and affecting academia and professional practice in many ways. Paradigms such as just-in-time learning, constructivism, student-centered learning and collaborative approaches have emerged and are being supported by technological advancements such as simulations, virtual reality and multi-agents systems. These developments have created both opportunities and areas of serious concerns.  

Editors of this special issue selected a number of papers presented at IADIS CELDA 2010 conference that were very highly rated by reviewers, well received at the conference, and nicely complementary in terms of research, theory, and implications for learning and instruction. These papers have been edited and revised based on feedback from conference participants and subsequent review by the editors of this special issue and reviewers recruited to assist in this process. The organizing committee of IADIS CELDA 2010 proposed a special issue of Educational Technology & Society Journal based on selected papers from IADIS CELDA 2010. The result is the five papers included in this special issue. 

As well as the five special papers, there are another 26 papers in this issue covering a diverse range of topics, including (at the post-secondary level):

  • Effects of Different Levels of Online User Identity Revelation
  • Student Satisfaction, Performance, and Knowledge Construction in Online Collaborative Learning: a cross-cultural perspective
  • A Context-Aware Mobile Learning System for Supporting Cognitive Apprenticeships in Nursing Skills Training
  • Exploring Non-traditional Learning Methods in Virtual and Real-world Environments
  • Providing Adaptivity in Moodle LMS Courses
  • Agent Prompts: Scaffolding for Productive Reflection in an Intelligent Learning Environment

Although the papers are in English, most of the authors are from either Eastern Europe or East Asia.

Online Educa Berlin 2011 program now available

Listen with webReader

The ONLINE EDUCA BERLIN 2011 programme is now available online. Featuring over 80 parallel sessions and 350 speakers, this year’s programme centres on New Learning Cultures.  Focusing on cutting-edge technologies and the latest policy developments, key questions in education and business will be addressed, including how can the delivery of education keep up with the pace of change? Do we need a new culture of learning? Are the old methods dead?

One of the keynote speakers is Ruth Martinez, E-Learning consultant and researcher in 3D Virtual Worlds. She is interviewed here:

Martinez, R. (2011) Virtual learning environments: an interview with Ruth Martínez Online Educa News Service, October 14

Another interesting presentation is about learner-directed learning. Thomas Köhler of the Institute for Vocational Education at Dresden University of Technology and Jens Drummer of the Saxony Education Institute have worked on a longitudinal study that looks at learner output in self-directed e-learning exercises. For more information, see:

Köhler, T. and Drummer, J. (2011) A learner-centred approach to Web 2.0 e-learning, Online Educa News Service, October 14

Where: Hotel Intercontinental, Budapester Str. 2, 10787 Berlin Germany Tel.: +49 (0)30 26 02-0

When: November 30-December 2