h1

Well, this is disappointing

April 1, 2013

Ben Goldacre responded on Twitter (@bengoldacre) to my blogposts (here and here) on his report.

Screenshot 2013-04-01 at 10.40.13 - EditedIf anyone can find anything I said to suggest RCTs were impossible to do in education, can you please identify it for me?

I’m actually of the view that:

  • RCTs can be used in education;
  • In some circumstances RCTs will be easy and convenient to do and it would be wrong not to do them;
  • RCTs should provide a higher standard of evidence than other methods.

However:

  • RCTs cannot resolve every issue;
  • RCTs will not persuade some of the key actors in education debates;
  • Other quantitative evidence should also be considered;
  • Their costs will not always be justified by their potential contribution to the evidence/debate;
  • Arguing over RCTs may distract from arguing in favour of evidence more generally.

These points (some of which apply to other methods too) don’t make RCTs impossible. They don’t even make them impractical. They simply make it absolutely vital that we first establish the principle of using evidence in education, and secondly, that we apply RCTs (and other methods) to the right questions. It will be interesting to see if the EEF work has done this, or whether it has simply applied RCTs to the pet projects of researchers; issues where there is already sufficent evidence, or to issues where one side will ignore the evidence regardless. It will also be interesting to judge the quality of the RCTs they have conducted and whether they will actually provide better evidence than other methods.

5 comments

  1. The key point here is one that you’ve mentioned many times before; we can’t agree what would constitute and intervention ‘working’. The common pre-test post-test format will never convince those who think that education is not about testing. This is not a marginal view at all and dome really big fish subscribe to it. I would also want to see prep work controlled in RCTs. If not, we’ll end up with inefficient schemes that keep teachers up into the small hours for marginal gains.


    • This! With bells on! At least in medicine a) all the subjects are broadly the same; and b) the key metric (is the subject still alive?) is easy to measure and impossible to game. No one has explained how RCTs could possibly work in practice in education, when the ‘subjects’ are (er…) subject to so many externalities you could never create a control group; and even if you could, we don’t even all agree on what the purpose is of education (economic? social? moral?), let alone how to measure its success in achieving it in a meaningful timespan.


  2. Hi

    I just think it’s odd that you’re talking about all these barriers as if RCTs in education were some kind of abstract business

    Why not discuss them concretely with examples, in reference to the EEF RCTs you regard as the *most* robust?

    Ben


  3. It would be well worth looking at the TCR (Columbia University) article by Dick Schutz – aneducational psychologist and psychometrist who has analysed educational testing for many decades. The article provides many examples where RCTs have not provided the answer. The RCT for SuccessforAll is particularly interesting – and depressing. It’s a pdf – I don’t have a link but google A Grand Experiment in Reading Instruction – TCR (Columbia University). Dr. Richard Schutz.
    If no success, email me (email should be back in working order shortly) –
    The concerns you raised are very legitimate.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: