October 31, 2014

Improving productivity in online learning: can we scale ‘the learning that matters most’?

Listen with webReader
Can 'the magic of the campus' be replicated online - and at scale?

Can ‘the magic of the campus’ be replicated online – and at scale?

The story so far

This is a continuation of the discussion on whether online learning can increase educational ‘productivity.’ Previous posts in this series include:

There is a CIDER webinar presentation on the HEQCO report available from here

In the last post, I concluded:

  • there are major economies of scale in using computer-based feedback for facilitating comprehension and technical mastery outcomes
  • computer-based feedback, when well designed, can also be useful in providing student feedback for more complex forms of learning, such as alternative strategies, critical thinking and evaluation
  • however computer-based analyses to date are inadequate for formal assessment of these higher order learning skills, where deep expertise and qualitative assessment is required, and where learners may provide new insights or alternative explanations
  • redesign of courses with a greater focus on student discovery (finding, analyzing and applying content) within a learning design offers more modest but still significant potential for increases in productivity, mainly through better learning outcomes (development of 21st century skills) and through more effective use of senior research professors’ time.

Learner-instructor interaction and economies of scale

In this current post, I examine particularly the learner-instructor interaction, and discuss whether online learning can provide economies of scale in this area. This is particularly important, because research on credit-based online learning has shown that course delivery (which includes both learner support and student assessment) accounted for the largest overall cost of an online program (37%), almost three times more than course development, over the life of an online program (Bates and Sangra, 2011).

Can we scale ‘the learning that matters most’?

This important question has been raised in the HEQCO report by Tom Carey and David Trick. It is this issue I wish to address here, since scaling up the delivery of content, and learner-content interaction, through online learning is relatively easy, although both depend on good course design for effective learning.

What is more challenging is whether we can also scale the kind of ‘learning that matters most’, namely helping students when they struggle with new concepts or ideas, helping students to gain deep understanding of a topic or subject, helping students to evaluate a range of different ideas or practices, providing students with professional formation or development, understanding the limits of knowledge, and above all enabling students to find, evaluate and apply knowledge appropriately in new or ill-defined contexts.

Before looking at whether or not such activities can be scaled, it is important to challenge the view, such as Sanjay Sharma’s at MIT, that such forms of learning can only be achieved on campus. There is also more than a hint of this assumption in the HEQCO report, at least with respect to undergraduate education. Those of us who have taught online will know that it is possible to develop these kinds of learning outcomes online, especially but not exclusively at graduate level. Strategies such as scaffolding or supporting knowledge construction through online discussion and dialogue, student reflection through e-portfolios, and above all personal online interventions and communication between students and instructor, have all been found to lead to learning outcomes at least as equivalent to those of students studying the same subjects on-campus (see references below).

There will remain a relatively few learning activities that matter most that are best done on campus, such as the development of hands-on skills, but there will be others, such as knowledge management, that may well be best done online. More importantly, there will be some students who really need the environment provided by a campus, and others that will prefer an online environment.

The issue is not can the learning that matters most be done online, but can it be scaled up through online learning? Certainly, I would argue that the main criticism of xMOOCs is that they spectacularly fail to address this form of learning. However, cMOOCs, when they operate at the level of communities of practice with relatively shared levels of understanding and knowledge among the participants, do have at least the potential for such economies of scale while maintaining or even improving quality of learning outcomes. The challenge though is how one accounts for the hidden costs of the participation of experts in such professional sharing, which rely heavily on volunteering or ‘moonlighting’ from a paid job by those with the expertise. I suspect though that even if these costs were calculated, they would still prove more ‘productive’ than conventional campus-based classes for this type of learner. However, the cost-effectiveness research has yet to be done.

The challenge though is scaling up the kinds of interaction between students and instructors that enable diagnosis of a student’s learning difficulties, that facilitate deep understanding of a subject, that encourage creative and original thinking, especially within undergraduate education. Adaptive learning and learning analytics may help to some extent, but in my view cannot yet come close to matching the skill of an experienced and skilled instructor. If instructors are to have enough time to engage in these kinds of dialogue and communication with students, there is clearly a limit on the number of students they can handle. Thus there is a possibility of small increases in productivity, aided by developments such as adaptive learning and learning analytics, but not major ones, in this aspect of teaching and learning.

Scaling the assessment of ‘learning that matters most.’

When ‘the magic of the campus’ is raised, one of the implicit assumptions is that student assessment is more valid because of the personal knowledge that faculty develop of a student in their entirety, and not just in their formal academic work: how they conduct themselves in class discussion (not just what they say, but how they say it), their interests and knowledge outside the formal curriculum (e.g. do they read widely or participate in valued extra-curricula activities), and the impression students make in social activities with faculty. This ‘tacit’ knowledge of a particular student that faculty acquire on campus can heavily influence the final assessment of a student, beyond that of the final exam. As they say at Oxford University, ‘Is he one of us?’

I was fortunate to have done my undergraduate degree in a department where every ‘honours’ student was well known by every faculty member. We were told that in the final exam, we could not get a worse grade than was already determined, but we could improve on it by a really good performance. In other words, the final exam was more of a rite of passage – the assessment was already more or less in place. This was only possible because of the ‘deep’ knowledge that faculty had already gained of the students. The fear that many faculty have of of online learning is that this kind of knowledge of a student is impossible ‘at a distance.’

Again, however, at least some elements of this ‘getting to know students’ can be achieved online, through continuous assessment, the use of e-portfolios and participation in online discussions. Again, the similarities between online learning and campus teaching are often greater than the differences. The problem is scaling up this kind of in-depth academic relationship between student and instructor, both for classroom and online teaching. Although the actual ratio may be difficult to specify, it is clear that this kind of relationship cannot be built up if the instructor:student ratio is in the thousands.

The fact is though that undergraduate students in most public universities are not in the fortunate position that I was. Even in their final year, many find themselves are in classes of over 100 students. They will probably be better off in an online class of 30 students, and even in an online class of 100, they may have more personal interaction with the instructor than in a lecture theatre, if the course is well designed. However, scaling up much beyond this ratio is not going to enable the more personal intellectual relationship to develop that allows for the more informal ‘I know what this student is capable of’ relationship, either online or on campus.

In short, for assessment based on deep knowledge of a student’s progress and capabilities, the scope for economies of scale are limited. In this sense, teacher:student ratios do matter, so economies of scale through online learning will be difficult to achieve for these higher order learning skills.

Conclusions

This has been a particularly difficult blog to write which suggests I may still not be thinking clearly about this topic, so please help me out! However, here is where I stand on this issue so far:

1. The ‘learning that matters most’ mainly addresses university teaching, but I suspect also increasingly technical, vocational and corporate training; the aim is to develop the knowledge and skills needed in a knowledge-based society.

2. Online learning can handle the ‘learning that matters most’ as well, in most cases, as on-campus teaching, although there will always be some exceptions.

3. However, there are major difficulties in scaling up the learner support and assessment activities that are needed for the learning that matters most, both online or on campus. The danger in scaling up is the loss of quality in terms of learning outcomes.

4. Adaptive learning software that helps individualize learning, and learning analytics, may help to a small degree in enabling instructors to handle slightly more students without loss of quality, but cannot as yet replace a skilled instructor, and probably never will.

5. New online course designs built around the use of new technologies have greater potential for increases in productivity – through producing better learning outcomes – for the learning that matters most, than through scaling up, i.e. by increasing teacher:student ratios.

6. We need more empirical research on the relationship between teaching methods, mode of delivery, costs, and the type of learning outcomes that constitute the ‘learning that matters most’ (not to mention better definitions).

Your input

First I’d really welcome responses to this post. In particular:

  • Is ‘the learning that matters most’ a useful concept for university teaching? Do you agree with my descriptions of it?
  • Have I missed something obvious in the possibility for scaling these learner support and assessment activities?
  • Can adaptive learning software and learning analytics take some or all of the load off instructors in developing such learning outcomes?
  • What would new online course designs that increase productivity look like? Do you have actual examples that have been implemented?

Next

In my next post on this topic, I will discuss an area where I think there is huge potential for increasing productivity through online learning, and that is through savings in physical overheads.

References

Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, Vol. 5, No.2.

Baker, C. (2010) The Impact of Instructor Immediacy and Presence for Online Student Affective Learning, Cognition, and Motivation The Journal of Educators Online Vol. 7, No. 1

Bates, A. and Sangrà, A. (2011) Managing Technology in Higher Education: Strategies for Transforming Teaching and Learning San Francisco: Jossey-Bass/John Wiley and Son

Garrison, D. R. & Cleveland-Innes, M. (2005). Facilitating cognitive presence in online learning: Interaction is not enough. American Journal of Distance Education, Vol. 19, No. 3

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Jonassen, D., Davidson, M., Collins, M., Campbell, J. and Haag, B. (1995) ‘Constructivism and Computer-mediated Communication in Distance Education’, American Journal of Distance Education, Vol. 9, No. 2, pp 7-26.

Paloff, R. and Pratt, K. (2007) Building Online Learning Communities San Francisco: John Wiley and Co.

Richardson, J. C., & Swan, K. (2003). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7 (1), 68-8 8.

Salmon, G. (2000) E-moderating London/New York: Routledge

Sheridan, K. and Kelly, M.  (2010) The Indicators of Instructor Presence that are Important to Students in Online Courses MERLOT Journal of Online Learning and Teaching, Vol. 6, No. 4

 

 

A bill of rights for online learners?

Listen with webReader

Morris, S.M. and Stommel, J. (2013) A Bill of Rights and Principles for Learning in the Digital Age Hybrid Pedagogy, January 22

I’ve just caught up with this (work keeps getting in the way of blogging, damn it) so forgive me if you’ve already seen it. This statement has been developed by a group meeting in Palo Alto, California, and has some well-known names attached, such as John Seeley Brown, Audrey Watters and Sebastian Thrun.

It’s really in two parts, the first setting out a collection of rights for learners and the second a statement of principles for providers of online learning. You will need to read the full article to get a more detailed description of each, but here is a very brief listing:

Rights (of learners)

  • to access: ‘Everyone should have the right to learn.’
  • to privacy
  • to create public knowledge
  • to own one’s own personal data and intellectual property
  • to financial transparency
  • to pedagogical transparency
  • to quality and care
  • to have great teachers
  • to be teachers

Principles (to which online learning should aspire)

  • global contribution: ‘Online learning should originate from everywhere on the globe, not just from the U.S. and other technologically advantaged countries.’
  • value: ‘The function of learning is to allow students to equip themselves to address the challenges and requirements of life and work.’
  • flexibility: ‘Ideally, they [the best online programs] will also suggest and support new forms of interdisciplinary and cross-disciplinary inquiry that are independent of old gatekeepers such as academic institutions or disciplines, certification agencies, time-to-degree measurements, etc.’
  • hybrid learning: ‘online learning should …. be connected back to multiple locations around the world and not tethered exclusively to the digital realm. ‘
  • persistence
  • innovation: ‘Online learning should be flexible, dynamic, and individualized rather than canned or standardized.
  • formative assessment
  • experimentation
  • civility
  • play

Comment

I have to admit being somewhat puzzled, not so much by the rights and principles themselves, but why it is thought necessary to codify and then publicize them.

First, would not most of these rights and principles be subscribed to already by most people that support public higher education, at least in North America, Europe and Australasia?  (I can’t speak for the Chinese or North Korean governments.)

If that’s the case (and it may be worth discussing this more), then the issue then is not the goals but the means to achieve the goals. Online learning is one, but in no way the only, means to some of these rights and principles. It is also true that while many working in or supporting public higher education would subscribe to these rights and principles, we often fall way short of implementing them, for a variety of reasons, such as lack of adequate resources or a poor choice of priorities. But that’s another discussion.

The question then comes to my mind as to why it has been necessary to spend time discussing and agreeing on principles and rights that most people in public education already accept.

One reason I suspect is a concern that developments in online learning outside formal, public education have the potential to run roughshod over these rights and principles. For instance, highly selective, campus-based, elite universities, at least until very recently, have not subscribed to some of these rights and principles, yet are now ‘discovering’ open learning through MOOCs, while still denying many of these rights to potential on-campus students.

Also, there is probably concern that MOOCs themselves are being exploited, at least by some organizations, for commercial reasons and this may result in some of these principles or rights being ignored or trampled on.

However, it could also be that some working in elite institutions have discovered God, and He is open, and so they need some commandments or a bible.

Thus having a statement of such rights and principles may be valuable, although how these rights or principles can be enforced is not at all clear to me – and what’s the use of a right if it can’t be protected?

Over to you

Do you think setting out these rights and principles is valuable?

Do you think public higher education generally subscribes to or adheres to these?

Why do you think such a statement has been made? Is it trying to say more than it does?

Don’t just tell me: join the conversation at https://twitter.com/search?q=%23learnersrights

See also: Kolowich, S. (2013)’Bill of Rights’ Seeks to Protect Students’ Interests as Online Learning Rapidly Expands Chronicle of Higher Education, January 23

 

 

IRRODL, Vol. 14, No. 1 now available

Listen with webReader

IRRODL (International Review of Research in Open and Distance Learning), Volume 14, Number 1 is now available, for free downloading as open educational resources.

This is a valuable pot-pourri of different topics, so it is not possible for me to do a review, but Terry Anderson provides an excellent editor’s summary of each of the articles.

Contents

Green curriculum: Sustainable learning in higher education HTML PDF MP3 EPUB
Willa Petronella Louw 1-15

 

A predictive study of student satisfaction in online education programs HTML PDF MP3 EPUB
Yu-Chun Kuo, Andrew E Walker, Brian R Belland, Kerstin E E Schroder 16-39

 

On-the-job e-learning: Workers’ attitudes and perceptions HTML PDF MP3 EPUB
Josep-Maria Batalla-Busquets, Carmen Pacheco-Bernal 40-64

 

An OER architecture framework: Need and design HTML PDF MP3 EPUB
Pankaj Khanna, P C Basak 65-83

 

Development of ODL in a newly industrialised country according to face-to-face contact, ICT, and e-readiness HTML PDF MP3 EPUB
J Marinda van Zyl, Christoffel Johannes Els, A Seugnet Blignaut 84-105

 

Employability in online higher education: A case study HTML PDF MP3 EPUB
Ana Paula Silva, Pedro Lourtie, Luisa Aires 106-125

 

Identifying barriers to the remix of translated open educational resources HTML PDF MP3 EPUB
Tel Amiel 126-144

 

Uses of published research: An exploratory case study HTML PDF MP3 EPUB
Patrick J. Fahy 145-166

 

A framework for developing competencies in open and distance e-learning HTML PDF MP3 EPUB
Patricia B Arinto 167-185

 

Peer Portal: Quality enhancement in thesis writing using self-managed peer review on a mass scale HTML PDF MP3 EPUB
Naghmeh Aghaee, Henrik Hansson 186-203

 

Learning in multiple communities from the perspective of knowledge capital HTML PDF MP3 EPUB
Hayriye Tugba Ozturk, Huseyin Ozcinar 204-221

 

A multimedia approach to ODL for agricultural training in Cambodia HTML PDF MP3 EPUB
Helena Grunfeld, Maria Lee Hoon Ng 222-238

 

Automatic evaluation for e-learning using latent semantic analysis: A use case HTML PDF MP3 EPUB
Mireia Farrús, Marta R. Costa-jussà 239-254

Field Notes

“Opening” a new kind of school: The story of the Open High School of Utah HTML PDF MP3 EPUB
DeLaina Tonks, Sarah Weston, David Wiley, Michael K. Barbour 255-271

Comment

This journal is possible only because of strong support from Athabasca University, which is undergoing some convulsive changes at the moment. If nothing else remains, I hope this journal survives, as it is an essential resource for those working in the field.

 

 

 

Who benefits from online learning?

Listen with webReader

Di Xu & Shanna Smith Jaggars (2013) Adaptability to Online Learning: Differences Across Types of Students and Academic Subject Areas New York: Community College Research Center, Columbia University

The study

Using a dataset containing nearly 500,000 courses taken by over 40,000 community and technical college students in Washington State, this study examines how well students adapt to the online environment in terms of their ability to persist and earn strong grades in online courses relative to their ability to do so in face-to-face courses. 

The hypothesis

Some populations of students—for example, those with more extensive exposure to technology or those who have been taught skills in terms of time-management and self-directed learning—may adapt more readily to online learning than others. In addition, some academic subject areas may lend themselves to highquality online learning experiences more readily than others

The methodology

Primary analyses were performed on a dataset containing 51,017 degree-seeking students who initially enrolled in one of Washington State’s 34 community or technical colleges during the fall term of 2004. These first-time college students were tracked through the spring of 2009 for 19 quarters of enrollment, or approximately five years.  The dataset, provided by the Washington State Board of Community and Technical Colleges (SBCTC), included information on student demographics, institutions attended, and transcript data on course enrollments and performance.

The results

  • In descriptive terms, students’ average persistence rate across courses was 94.12 percent, with a noticeable gap between online courses (91.19 percent) and face-to-face courses (94.45 percent). For courses in which students persisted through to the end of the term (N = 469,287), the average grade was 2.95 (on a 4.0-point scale), also with a gap between online courses (2.77) and face-to-face courses (2.98).
  • While all types of students in the study suffered decrements in performance in online courses, some struggled more than others to adapt: males, younger students, Black students, and students with lower grade point averages. 
  • Regardless of a particular student’s own adaptability to the online environment, her performance in an online course may suffer if her classmates adapt poorly. English and social science were two academic subjects that seemed to attract a high proportion of less-adaptable students, thereby introducing negative peer effects.
  • Older students adapted more readily to online courses than did younger students.
  • [Also] students who were more disposed to take online course also tended to have stronger overall academic performance than their peers  

 

Comments

First, it is encouraging to see a detailed quantitative assessment of the types of students taking online courses, and their relative performance. This report needs to be read in full, and carefully. It is good that it is based on a significantly large enough sample that one can have confidence in the generalizability of the results (at least in the U.S. two-year college sector). The study was very well carried out and is a model for quantitative analysis of student differences.

Furthermore, I am not surprised or even concerned about these findings. For instance, from my own experience of online teaching, I would agree that ‘students are not homogeneous in their adaptability to the online delivery format and may therefore have substantially different outcomes for online learning.’ Online learning doesn’t suit everyone, and it is valuable to have some research that helps identify the more ‘at-risk’ online learners.

One can put forward a number of reasons why online students, on average, are likely to struggle compared with face-to-face students. Students who choose an online course are likely on average to have less time for study that those attending regularly on campus. Second, for many online students, the mode of study will be unfamiliar, which means making more adaptation to a different way of learning.

My one quibble is that, although the results are clearly significant statistically (as is almost inevitable in large samples), the differences are quite small (96% vs 91% completion rates, for instance.) Thus I do challenge the authors’ conclusion that ‘most students had difficulty adapting to the online context.’ If 91% complete the course, then most students did not have difficulties sufficient to deter them from completing successfully their courses. That seems a pretty good adaptation level to me.

I also have a concern that these results will be misinterpreted. This should not mean that men, Blacks and young people should be discouraged from taking online courses, but that we should be taking more care to ensure that students who do take online courses are better prepared, with particular attention being paid to those likely to be at most risk. This may mean, for instance, gradually introducing students to online learning in a deliberate way throughout a program. This study does suggest those most likely to be at risk.

Further reading

Lederman, D. (2013) Who benefits from online Ed? Inside Higher Education, February 25

 

No. 1 aha moment: media are different

Listen with webReader

© Open University 2013

In a previous post, I listed the seven ‘aha’ moments that have been the most seminal ‘discoveries’ in my researching and working in educational technology. This is the first of seven posts that discusses why I believe these ‘discoveries’ to be important, and their implications specifically for online learning.

What was the discovery?

Different media have different educational effects or affordances. If you just transfer the same teaching to a different mediium, you fail to exploit the unique characteristics of that medium. Put more positively, you can do different and often better teaching by adapting it to the medium. That way students will learn more deeply and effectively.

How did this discovery come about?

Aha moments do not come out of thin air. A number of things come together until something in particular triggers it.

The first one, media are different, came very early, within 12 months of starting my career as a researcher into educational technologies. In 1969, I was appointed as a research officer at the Open University in the United Kingdom. At this point the university had just received its royal charter. I was the 20th member of staff appointed. My job was simple: to research into the pilot of the OU being currently offered by the National Extension College, which was offering low cost non-credit distance education programs in partnership with the BBC. ( So you think MOOCs are new?! The NEC was offering them over 40 years ago).

We sent out by mail questionnaires on a weekly basis to students taking these courses. The questionnaire contained both pre-coded responses, and the opportunity for open-ended comments, and asked students for their responses to the print and broadcast components of the courses. We were looking for what worked and what didn’t work in designing multimedia distance education courses.

When I started analyzing the questionnaires, I was struck particularly by the ‘open-ended’ comments in response to the television and radio broadcasts. Responses to the printed components tended to be ‘cool': rational, calm, critical, constructive. The responses to the broadcasts were the opposite: emotional, strongly supportive or strongly critical or even hostile, and rarely critically constructive. Something was going on here.

Since the OU was going to spend 20% of ita annual budget on the broadcasts from the BBC, I persuaded the university to appoint me as a lecturer to research into the effectiveness of the television and radio programs, which I did for a period of nearly 20 years.

The initial discovery that media were different came very quickly, but it took longer to discover in what ways media are different, and even longer why. I have written more extensively about this elsewhere, but here are some of the interesting discoveries I and my colleagues in the Audio-Visual Media Research Group at the OU made:

  • the BBC producers (all of whom had a degree in the subject area in which they were making programs) thought about knowledge differently from the academics with whom they were working. In particular, they tended to think more visually and more concretely about the subject matter. Thus they tended to make programs that showed concrete examples of concepts or principles in the texts, applications of principles, or how academic concepts worked in real life.
  • the BBC producers rarely used talking heads or TV lectures. With radio and later audio-cassettes, some producers and academics integrated the audio with texts, for instance in mathematics, talking students through equations or formulae in the printed text (similar to Khan Academy lectures on TV)
  • students responded very differently to the TV programs in particular. Some loved them, some hated them, and few were indifferent. The ones that hated them wanted the programs to be didactic and repeat  or reinforce what was in the printed texts. They tended to get lower grades or even fail in the final course exam. The ones that loved them tende to get higher grades. They were able to see how the programs illustrated the principles in the texts, and the programs tended to ‘stretch’ students to think more widely or critically about the topics in the course. The exception was math, where borderline students found the TV programs most helpful
  • using television and radio to develop higher level learning is a skill that can be taught. In the foundation social science course (D100), many of the programs were made in a typical BBC documentary style. Although the programs were accompanied by extensive broadcast notes that attempted to link the broadcasts to the academic texts, many students struggled with these programs. When the course was remade five years later a distinguished academic (Stuart Hall) was used as an ‘anchor’ for all the programs. The first few programs were somewhat like lectures, but in each program Stuart Hall introduced more and more visual clips and helped students analyze each clip . By the end of the course the programs were almost entirely in the documentary format. Students rated the remade program much higher and used examples from the TV programs much more in their assignments and exams for the remade course.

Why are these findings significant?

At the time (and for many years afterwards) researchers such as Richard Clark (1983) argued that the research showed no significant different between the use of different media. In particular, there were no differences between classroom teaching and other media such as television or radio or satellite or the Internet. Even today, we are getting similar findings regarding online learning (e.g. Means et al. 2010)

However, this is because  the research methodology that is used by researchers for such comparative studies requires the two conditions being compared to be the same, except for the medium being used. Therefore a classroom lecture had to be compared to a television lecture. Indeed Clark argued that any differences were due to pedagogical differences in the media use. Since the classroom was used as the base, you had to strip out all the affordances of television – what it could do better than a lecture – in order to compare it.

The critical point is that different media can be used to assist learners to learn in different ways and achieve different outcomes. In a sense, researchers such as Clark were right: the teaching methods matter, but different media can more easily support different ways of teaching than others.

Perhaps even more important is the idea that many media are better than one. This allows learners with different preferences for learning to be accommodated, and to allow subject matter to be taught in different ways through different media, thus leading to deeper understanding or a wider range of skills in using content.

How does this apply to online learning?

Online learning can incorporate a range of different media: text, graphics, audio, video, animation, simulations. We need to understand better their affordances, and use them differentially so as to develop deeper knowledge, and a wider range of learning outcomes and skills.

The use of different media also allows for more individualization and personalization of the learning, better suiting learners with different learning styles and needs.

Most of all, we should stop trying merely to move classroom teaching to other media such as MOOCs, and start designing online learning so its full potential can be exploited.

References and further reading

Bates, A. (1985) Broadcasting in Education: An Evaluation London: Constables (out of print – try a good library)

Bates, A. (2012) Pedagogical roles for video in online learning, Online Learning and Distance Education Resources

Clark, R. (1983) ‘Reconsidering research on learning from media’ Review of Educational Research, Vol. 53, pp. 445-459

Kozma, R. (1994) ‘Will Media Influence Learning? Reframing the Debate’, Educational Technology Research and Development, Vol. 42, No. 2, pp. 7-19

Means, B. et al. (2009) Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies Washington, DC: US Department of Education (http://www.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf)

Russell, T. L. (1999) The No Significant Difference Phenomenon Raleigh, NC: North Carolina State University, Office of Instructional Telecommunication