August 14, 2018

A report on online learning and educational productivity: disappointed!!!!

Kevin Kline and Michael Palin in A Fish Called Wanda

Bakia, M., Shear, L. Toyama, Y. and Lasseter, A. (2012) Understanding the implications of online learning for educational productivity Washington DC: Department of Education Office of Educational Technology

This report was done for the US Department of Education by SRI International, a non-profit research and development organization. The report’s focus is on secondary schools in the USA.

This is the most important finding:

Unfortunately, a review of the available research that examined the impact of online learning on educational productivity for secondary school students was found to be lacking. No analyses were found that rigorously measured the productivity of an online learning system relative to place-based instruction in secondary schools.

The evidence summarized in this report draws on literature that addressed either costs or effectiveness. These studies typically were limited because they did not bring the two together in a productivity ratio and compare results with other alternatives.

The report then states:

Given the limitations of the research regarding the costs and effects of online instruction for secondary students, the review that follows also draws on examples and research about the use of online learning for postsecondary instruction.

I eagerly awaited this report as there is a huge controversy in the USA at the moment over charter online schools which is resulting in a lot of negative press for online learning, as online learning and privatization are being linked together. Some hard facts about the costs of online learning, especially between public and private charter schools, and the learning outcomes, would be really valuable. However, as the authors note, you have to compare like with like, and none of the studies they explored did this (although the studies they looked at consistently showed lower cost for online schools).

Although there is absolutely no empirical evidence or data in the report to back this, the authors came to the following conclusions:

Nine applications of online learning …are seen as possible pathways to improved productivity:

  1. Broadening access
  2. Engaging students in active learning
  3. Individualizing and differentiating instruction
  4. Personalizing learning
  5. Increased student motivation, time on task and ultimately better learning outcomes;
  6. Making better use of teacher and student time by automating routine tasks and enabling teacher time to focus on high-value activities;
  7. Increasing the rate of student learning by increasing motivation and helping students grasp concepts and demonstrate competency more efficiently;
  8. Reducing school-based facilities costs by leveraging home and community spaces in addition to traditional school buildings;
  9. Reducing salary costs by transferring some educational activities to computers, by increasing teacher-student ratios or by otherwise redesigning processes that allow for more effective use of teacher time; and
  10. Realizing opportunities for economies of scale through reuse of materials and their large-scale distribution

Although I would agree with most of these, these are arguments not evidence, and could be made by any informed proponent of online learning.

If you are still with me by now, there are basically three main conclusions that can I draw from this report:

  • the authors found no usable evidence by their own standards and hence have no basis for their conclusions
  • The US Department of Education wasted its money: there is nothing new in this report that wasn’t known already
  • it’s not a good idea to set up such a rigorous standard for the design of research that the research can’t be done – especially if the taxpayer is paying for it.

So: my reaction is the same as Kevin Kline’s in A Fish Called Wanda when he opened the safe and found it empty: ‘DISAPPOINTED!’

The main outcome may be that the Department of Education might now fund some real research on this important topic, but the Department is now no doubt well aware that this is very difficult research to do well. The main reasons are that online learning, like face-to-face teaching, can cost what you want it to cost, and it is difficult to put a meaningful price on many of the important benefits, such as increased access. The really difficult part though would be comparing learning outcomes, but this could be done, even if only on standardized scores, as unsatisfactory as that may be. But then we already know the likely result: no significant difference, when all other variables are controlled, which they never are in education.

My advice would be not to do a classic cost-benefit analysis because it won’t likely provide meaningful results, but take a more qualitative case-study approach that looks at the specific pros and cons of online learning in specific cases, then draw some general conclusions from this about the relationship between costs and results. Then you might get results of this kind: saving on teachers’ costs in online learning is not a good idea because the quality goes down, but access is increased at lower cost than building a physical school. But I guess such results would be too complicated for politicians and advocates who want a yes/no answer.

Hints on a national strategy for educational technology in the USA

Downey, G. (2009) ED’s new tech chief previews national plan eSchool News, December 2

This is a report of a recent speech by Karen Cator, the new director of education technology at the US Department of Education. The new National Education Technology Plan will have four areas of focus: access, assessment, learning support, improving cost-effectiveness/productivity.

Cator said that the U.S. Department of Education (ED) will unveil the first draft of the administration’s National Education Technology Plan next month.

Suggestions for a national ed tech plan for the USA

eSchool News (2009) Stakeholders advise on national ed-tech plan eSchool News, November 17

U.S. Department of Education (ED) is preparing a new National Education Technology Plan and is accepting public feedback as it develops the new plan. A department spokesman did not know how long the public comment period would remain open, saying only that ED hopes to release the new plan in early 2010. More than 200 comments had been submitted to, the new plan’s web site.

This article discusses recommendations from Susan Patrick, the person who oversaw creation of the last national ed-tech plan in 2004.Her submission states; “Online course enrollments are growing at 30 percent annually, but fall short of [student] demands. National surveys show that the [percentage] of [middle and high school] students [who] are interested in taking an online course is 40 percent. … [That’s] far more than the 2 million enrollments today. Outdated laws, policies, teacher preparation, professional development, and funding models limit student choices … made possible through online learning.”

Her recommendations include:

  • A call for teacher-education programs at colleges and universities to train every pre-service teacher to teach online.

  • Provide incentives for portability of credits among institutions and across state lines to support virtual learning.

  • Provide incentives for true reciprocity of teacher professional licensure for online teaching.

  • Encourage states to expand online-learning opportunities for students by making this a requirement for federal innovation and ed-tech funding.

  • Invest in the development of open courses and curriculum with federal and state funding.

  • Finish the job of ensuring ubiquitous internet access in schools and at home.

This eSchool article also includes discussion of submissions from other organizations, such as the Creative Commons and Sesame Street

Now wouldn’t it be nice if there was a similar initiative from the Canadian Federal Government for Canadian universities – oh, I forgot, education is a provincial responsibility, and universities are autonomous, eh?

Higher education students in the USA in 2018

Hussar, W. and Bailey, T. (2009) Projections of education statistics to 2018 US Department of Education National Center for Educational Statistics: Washington DC

The education department’s projections for schools granting degrees show enrollment will increase:

  • 38% for Hispanic students, 26% for black students and 4% for white students
  • 16% for women and 9% for men
  • 9% for students ages 18-24; 25% for students 25-34; 12% for students 35 and older
  • 12% for undergraduates and 18% for graduate students
  • to 20.6 million overall in fall 2018 – 13% higher than 2007 stats

Some implications

1. Increase of 13% in overall student numbers will put even more pressure on funding. My guess: even larger classes if there is no radical change in teaching/institutional delivery methods

2. Increase in proportion of older and graduate students will increase demand for more online learning, flexible delivery, and distance courses/programs

3. Increasing diversity of students will require more diverse/adaptable teaching strategies to accommodate increasing differences in learning style’s/academic abilities.

How will institutions respond to this? Larger face-to-face classes or more intelligent use of technology? Answers on a postcard please (actually, comment box below will do).

Online students perform better than face-to-face students

Means, B. et al. (2009) Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies Washington, DC: US Department of Education

From the Abstract

A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning. Analysts screened these studies to find those that (a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size. As a result of this screening, 51 independent effects were identified that could be subjected to meta-analysis. The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction.

That seems pretty conclusive, for a research report, doesn’t it? Well, note the following ‘caveat’ in the report:

“In many of the studies showing an advantage for online learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages. At the same time, one should note that online learning is much more conducive to the expansion of learning time than is face-to-face instruction.”

The report also noted that blended learning (a combination of face-to-face and online learning) produced the best results, but this may be mainly due to the extra ‘time-on-task’ of students in such a context.

For further comments on this report, see:

Jaschik, S. (2009) The Evidence on Online Education Inside Higher Education, June 29

Terry Anderson’s blog: