Universitat Oberta de Catalunya (2016) An Adaptive Trust-based e-assessment system for learning (@TeSLA) Barcelona: UOC
This paper describes a large, collaborative European Commission project headed by the Open University of Catalonia, called TeSLA, (no, not to develop a European electric car, but) a state-of-the-art online assessment system that will be accepted as equal to if not better than traditional face-to-face assessment in higher education.
The project argues that at the moment there is no (European?) online assessment system that:
- has the same level of trust as face-to-face assessment systems
- that is universally accepted by educational institutions, accreditation agencies and employers
- incorporates pedagogical as well as technical features
- integrates with other aspects of teaching and learning
- provides true and secure ‘authentication’ of authorship.
I added the ‘European’, as I think this claim might come as a surprise to Western Governors’ University, which has been successfully using online proctoring for some time. It is also why I used the term ‘next generation’ in the heading, as the TeSLA project is aiming at something much more technologically advanced than the current WGU system, which consists mainly of a set of web cameras observing learners taking an assessment (click here for a demonstration).
Also, the TeSLA proposal makes a good point when it says any comprehensive online assessment system must also be able to handle formative as well as summative assessment, and that this can be a challenge as formative assessment is often embedded in the day-to-day teaching and learning activities.
But the main reason for this project is that online learning assessment currently lacks the credibility of face-to-face assessment.
A non-invasive system that is able to provide a quality continuous assessment model, using proportionate and necessary controls that will ensure student identity and authorship [in a way that offers] accrediting agencies and society unambiguous proof of academic progression….
Any solution must work fully online and take into account ‘academic requirements’ for assessment, including enriched feedback, adaptive learning, formative assessment and personalized learning.
This will require the use of technologies that provide reliable and accurate user authentication and identification of authorship, face and voice recognition, and keystroke dynamics recognition (see here for video examples of the proposed techniques).
The solution must result in
a system based on demonstrable trust between the institution and its students. Student trust is continuously updated according to their interaction with the institution, such as analysis of their exercises, peer feedback in cooperative activities or teacher confidence information. Evidence is continuously collected and contrasted in order to provide such unambiguous proof.
The participants in this project include
- eight universities,
- four research centres,
- three educational quality assurance agencies,
- three technology companies,
- from twelve different countries.
In total the project will have a team of about 80 professionals and will use large-scale pilots involving over 14,000 European students.
I think this is a very interesting project and is likely to grab a lot of attention. At the end of the day, there could well be some significant improvements to online assessment that will actually transfer to multiple online courses and programs.
However, I spent many years working on large European Commission projects and I am certainly glad I don’t have to do that any more. Quite apart from the truly mindless bureaucracy that always accompanies such projects (the form-filling is vast and endless), there are real challenges in getting together participants who can truly contribute to such a project. Participants are determined more by political considerations, such as regional representation, rather than technical competence. Such projects in the end are largely driven by two or three key players; the remaining participants are more likely to slow down or inhibit the project, and they certainly divert essential funding away from the those most able to make the project succeed. However, these projects are as much about raising the level of all European countries in terms of learning technologies as becoming a world leader in this field.
These criticisms apply to any of the many European Commission projects, but there are some issues that are particular to this project:
- I am not convinced that there is a real problem here, or at least a problem that requires better technology as a solution. Assessment for online learning has been successfully implemented now for more than 20 years, and while it mostly depends on some form of face-to-face invigilation, this has not proved a major acceptability problem or a barrier to online enrolments. There will always be those who do not accept the equivalence of online learning, and the claimed shortcomings of online assessment are just another excuse for non-acceptance of online learning in general.
- Many of the problems of authenticity and authorship are the same for face-to-face assessment. Cheating is not exclusive to online learning, nor is there any evidence that it is more prevalent in online learning where it is provided by properly accredited higher education institutions. Such a study is just as likely to reduce rather than increase trust in online learning by focusing attention on an issue that has not been a big problem to date.
- Even if this project does result in more ‘trustworthy’ online assessment, there are huge issues of privacy and security of data involved, not to mention the likely cost to institutions. Perhaps the most useful outcome from this project will be a better understanding of these risks, and development of protocols for protecting student privacy and the security of the data collected for this purpose. I wish though that a privacy commissioner was among the eighteen different participants in this project. I fail to see how such a project could be anything but invasive for students, most of whom will be assessed from home.
For all these reasons, this project is well worth tracking. It has the potential to radically change the way we not only assess online learners, but also how we teach them, because assessment always drives learner behaviour. Whether such changes will be on balance beneficial though remains to be seen.