Kevin Kline and Michael Palin in A Fish Called Wanda

Bakia, M., Shear, L. Toyama, Y. and Lasseter, A. (2012) Understanding the implications of online learning for educational productivity Washington DC: Department of Education Office of Educational Technology

This report was done for the US Department of Education by SRI International, a non-profit research and development organization. The report’s focus is on secondary schools in the USA.

This is the most important finding:

Unfortunately, a review of the available research that examined the impact of online learning on educational productivity for secondary school students was found to be lacking. No analyses were found that rigorously measured the productivity of an online learning system relative to place-based instruction in secondary schools.

The evidence summarized in this report draws on literature that addressed either costs or effectiveness. These studies typically were limited because they did not bring the two together in a productivity ratio and compare results with other alternatives.

The report then states:

Given the limitations of the research regarding the costs and effects of online instruction for secondary students, the review that follows also draws on examples and research about the use of online learning for postsecondary instruction.

I eagerly awaited this report as there is a huge controversy in the USA at the moment over charter online schools which is resulting in a lot of negative press for online learning, as online learning and privatization are being linked together. Some hard facts about the costs of online learning, especially between public and private charter schools, and the learning outcomes, would be really valuable. However, as the authors note, you have to compare like with like, and none of the studies they explored did this (although the studies they looked at consistently showed lower cost for online schools).

Although there is absolutely no empirical evidence or data in the report to back this, the authors came to the following conclusions:

Nine applications of online learning …are seen as possible pathways to improved productivity:

1. Broadening access
2. Engaging students in active learning
3. Individualizing and differentiating instruction
4. Personalizing learning
5. Increased student motivation, time on task and ultimately better learning outcomes;
6. Making better use of teacher and student time by automating routine tasks and enabling teacher time to focus on high-value activities;
7. Increasing the rate of student learning by increasing motivation and helping students grasp concepts and demonstrate competency more efficiently;
8. Reducing school-based facilities costs by leveraging home and community spaces in addition to traditional school buildings;
9. Reducing salary costs by transferring some educational activities to computers, by increasing teacher-student ratios or by otherwise redesigning processes that allow for more effective use of teacher time; and
10. Realizing opportunities for economies of scale through reuse of materials and their large-scale distribution

Although I would agree with most of these, these are arguments not evidence, and could be made by any informed proponent of online learning.

If you are still with me by now, there are basically three main conclusions that can I draw from this report:

• the authors found no usable evidence by their own standards and hence have no basis for their conclusions
• The US Department of Education wasted its money: there is nothing new in this report that wasn’t known already
• it’s not a good idea to set up such a rigorous standard for the design of research that the research can’t be done – especially if the taxpayer is paying for it.

So: my reaction is the same as Kevin Kline’s in A Fish Called Wanda when he opened the safe and found it empty: ‘DISAPPOINTED!’

The main outcome may be that the Department of Education might now fund some real research on this important topic, but the Department is now no doubt well aware that this is very difficult research to do well. The main reasons are that online learning, like face-to-face teaching, can cost what you want it to cost, and it is difficult to put a meaningful price on many of the important benefits, such as increased access. The really difficult part though would be comparing learning outcomes, but this could be done, even if only on standardized scores, as unsatisfactory as that may be. But then we already know the likely result: no significant difference, when all other variables are controlled, which they never are in education.

My advice would be not to do a classic cost-benefit analysis because it won’t likely provide meaningful results, but take a more qualitative case-study approach that looks at the specific pros and cons of online learning in specific cases, then draw some general conclusions from this about the relationship between costs and results. Then you might get results of this kind: saving on teachers’ costs in online learning is not a good idea because the quality goes down, but access is increased at lower cost than building a physical school. But I guess such results would be too complicated for politicians and advocates who want a yes/no answer.