March 23, 2011

How good is education research? (Post 1 of 2)

An article in the November 2010 Atlantic (yes I’m a bit behind on my reading), titled Lies, Damn Lies, and Medical Science, got my attention and made me wonder if there are corollaries to education research.

First, the article background: it details work by a meta-researcher, John Ioannidis, who has shown that a distressingly high proportion of medical studies published in journals are wrong. In one example, he looked at 49 of the most-cited articles in journals that are themselves most cited.[1] He found that the research in 35 of the studies had been retested, and 14 of them (41%) were subsequently shown to be either incorrect or highly exaggerated. This is just one example, and the article makes a strong case that medical science is often not on the solid ground that it appears when your doctor prescribes a certain drug for your condition. Research suffers from the following issues, among others:

  • – Studies showing an important new finding are much more likely to be published than studies that don’t find something new. This creates an unconscious bias in researchers to find positive results.
  • – Studies refuting earlier findings are much less likely to be prominently reported than the original studies that have been widely disseminated. The result is that doctors continue to cite the original, refuted article and findings.
  • – So many factors impact patients’ health that teasing out the effect of one factor is extremely difficult, and even if an impact is found with some patients it may not apply to others.

The article is well worth a read on its own merits, and it made me think of the many ways in which “education research” could be substituted for “medical research.” I’ve come to believe three things about education research:

  • – Doing randomized controlled experiments is so time-consuming and expensive that most research, even widely cited research, does not reach this standard.
  • – Even the gold standard has drawbacks (Ioannidis believes that 25% of randomized controlled experiments in medicine are wrong).
  • – The subtle explanations and caveats that are often explained in the studies are not well represented by media accounts of the research.

What does this mean for online learning? I’ve not yet determined what I think the ultimate answer is to that question. My interim sense is that the implications include:

1. We should be careful about citing studies, particularly ones that we haven’t read ourselves.

2. Foundations and policymakers should temper their expectations about how quickly research into online learning results will be available, and how quickly a body of evidence will have been created.

3. A lack of empirical studies is not reason to hold back on creating online learning options.

More on item #3 in the next blog post.

I’m sure there are some education researchers with a more robust perspective than mine, and I look forward to hearing from them.

[1] This approach to rating articles and citations is common across fields of science.

2 Responses to How good is education research? (Post 1 of 2)

  1. John, this is nothing new. My own doctoral advisor gave a keynote at the AECT conference in 1995 (see ) arguing much the same point. He’s refine that discussion in a new article recently published at

    Before Tom began talking about this, Richard Clark did a wondering review in 1983 about the impact – or lack thereof – of media comparison research (essentially the randomized control experiments you reference above, which the previous administration mistakenly promoted as the only way to do rigourous research). Clark’s Review of Educational Research article was simply the latest – at the time – of a long line of similar articles.

    I’ve actually presented on this very topic that you are discussing in this entry (and I think what may be coming in Part Two). I’ve been toying with the idea of crafting my own blog entry or doing a new podcast – something I have been neglecting lately – on the issue.

    • Michael, thanks for your comment. Whether or not this view is new to you and other researchers, I guarantee that it is not a commonly held belief among policymakers. We routinely get questions about research supporting online and blended learning, which leads (in my view) to the overuse of a few limited studies. To the extent that we can get discussions like this out of the research literature, and into broader public discussions, particularly with policymakers, that would greatly benefit the political and policy discussions that are taking place in many states.

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

WordPress SEO fine-tune by Meta SEO Pack from Poradnik Webmastera