A probabilistic approach to prediction is wise

Before drawing up my outlook for 2013, I want to discuss the important topic of prediction in online learning, in particular how predictions are made, and what value they may have. Nate Silver’s excellent book (references are at the end of this article) looks at prediction in a number of fields: weather forecasting (excellent up to three days, useless after eight days), economic forecasting (hopeless by both media pundits and professional economists), baseball players’ performance (pretty good and improving), earthquakes (bad for major quakes, but promising for lesser quakes), poker and a number of other areas. He also has some interesting reflections on big data as well. Unfortunately though he doesn’t discuss prediction in online learning, so I’ll try and help out with this!

Factors associated with reliable predictions

Silver’s book is valuable because he sets out some of the factors associated with good prediction (or forecasting):

1. Well understood and empirically supported theory about what drives the field under inquiry (excellent in weather forecasting and earthquakes; poor in politics and economics)
2. Large, reliable sets of relevant data and the ability to crunch large data sets
3. Relatively stable movement within the data (i.e. not too much ‘noise’ or randomness)
4. Elimination of or accounting for as far as possible the unknown
5. Above all, a probabilistic approach to prediction that takes account of uncertainty.

Factors associated with online learning

The problem for online learning is that few of these factors exist. In terms of theory, we do have some some empirically supported theories about what makes for effective online learning (e.g. Linda Harasim’s Learning Theory and Online Technologies) and some standards for best practices. However, these are often not practiced, or are ignored, in the field of online learning, and more importantly we lack good, empirically based theories of organizational decision-making in post-secondary education. This makes the application of what theory we have to understanding data and looking for the signal in the noise particularly hazardous for online learning.

The situation is even worse with regards to data. Weather forecasting data is detailed, localized and goes back over 60 years. Online learning is itself barely 20 years old (at least as we now know it), and is continually changing (as is the weather of course, but at least meteorologists know why the weather changes).

We have very little data on what is actually happening in online learning, and over-reporting in some areas (e.g. MOOCs) and under-reporting in others (for-credit programs). We are almost entirely dependent on the Sloan/Babson annual surveys for online learning enrollments and the Kenneth Green survey for IT developments on campuses, both covering just the USA. These surveys are invaluable, especially because they use a consistent methodology from year to year, enabling comparisons to be made, but they depend on the voluntary participation of selected staff within institutions, which tends to provide a bias to over-reporting online activities. In Canada, we have nothing, except a 2010 survey in Ontario which is unlikely to be repeated. So the statistical basis for reliable prediction in online learning just isn’t there.

With regard to ‘unknowns’ in online learning, they are everywhere but of course not visible until they hit you. MOOCs are a good example of something suddenly jumping out of the bush at you. But we have had other scares as well, such as for-profit universities. And some of the scares or unknowns quickly become very real in online learning, while others disappear almost as quickly as they came.

Fairy stories are perhaps a better basis for understanding online learning than scientific prediction. This is how I see MOOCs

The last factor though, a more probabilistic approach, is one we can apply to online learning. Silver makes the distinction between hedgehogs – pundits who have a strong view on everything, a ‘biased’ or strong ideological position, and who tend to make statements with a high degree of certainty, but who are frequently and routinely completely wrong- and foxes, who tend to be more cautious in their statements, are more equivocal in their predictions, but in the long haul have a better track record of accurate predictions. Foxes take a more probabilistic approach, recognizing degrees of uncertainty in their predictions (not necessarily in mathematical terms).

Timing as a factor in online learning predictions

A particular problem with prediction in online learning is the timing. The Horizon reports deal with this by having one, three and five year projections which is a more probabilistic approach, but I would argue it is more of a hedgehog than a fox because it focuses mainly on technology and not on pedagogy, and usually does not hedge its bets. Jon Baggaley, in a forthcoming analysis of the Horizon reports, also shows how unreliable their predictions have been.

In online learning, technology moves faster than people, and people move faster than organizations. So where you see changes in individuals, it may be another 10 years before that filters through to true organizational adoption. Also when does a prediction become true? Let’s take hybrid learning. Does 100 instructors moving to hybrid learning constitute a ‘trend’? However, if 100 institutions move in that direction within a year, that would be more significant. So as well as timing, the level of analysis matters too.

Why prediction in online learning is still necessary

Audrey Watters, who is my favourite blogger on online learning and educational technology, has also read Nate Silver’s book, and is aware of many of the problems I have just laid out. For these reasons, she has decided not to make any predictions for 2013. She is no doubt wiser than me, but I think it’s a pity she’s opting out. She is in a much better position than most of us to make predictions about online learning because she has a very broad overview, a full picture, of what is going on, even if the details are not always clear.

The fact is, we have to make predictions, every day in our lives – is it going to rain (so I take an umbrella), will the stock market go down (so I won’t invest $5,000), will my house still have enough equity when I need to go into an old folk’s home (yes, so I’ll go on holiday this year), will my bosses want to do MOOCs (yes, so I’d better be prepared)? And we always have to make these predictions with biases, less than perfect data, and lots of nasty unknowns lurking in the garden.

Silver’s book in fact does not argue against predictions, but doing them as well as possible. You do the best you can, and take a probabilistic approach. (If I don’t use the umbrella, and it doesn’t rain, no big deal, I’ll wait two months before reconsidering my investment, I’ll chose a budget holiday, I’ll suggest a way to do MOOCs that enhances their quality). We will all have to make some predictions, some intelligent guesses, as to what’s going to happen in online learning this year, so we can at least be prepared.

Go for it, baby

I will be making some predictions, not because I have all the data I’d like (you never do, even in meteorology). I also have my biases and prejudices. However, I do have a lot of experience in online learning, which provides at least some sort of theoretical framework for analysis, I do get to see what’s happening in about 10-15 universities and colleges a year (not enough but more than many), I do read a lot of the research literature on online learning, and I cover a huge amount of news and developments for my blog. So you decide whether or not my predictions are likely to be better than yours. At least you can make a comparison. (Silver points out that the average of multiple sources of predictions is usually more accurate than single sources of prediction, so let’s all share).

So, yes, you will get an outlook for 2013 for online learning from me. I will make some firm predictions, but I will use a one to five year horizon, and there will be caveats, and the unknowns will still jump out at you during the year – but at least you’ll have an umbrella to fend them off, and you can then blame me if it all goes wrong.

Silver, N. (2012) The Signal and the Noise: Why So Many Predictions Fail - but Some Don’t New York: The Penguin Press

Watters, A. (2013) Why I’m not making Ed-Tech predictions for 2013 Hack Education, January 1

Baggaley, J. (in press) Shifting Horizons, Distance Education