Policy Based Evidence MakingFebruary 8, 2013
Apologies for being a week late (and, therefore, not terribly topical) with this one. There were technical difficulties.
There seems to have been a craze recently for people engaged in partisan arguments about education to claim that those they disagree with have ignored the evidence. It seems that the more ideological one’s own position is, the more likely it is that one will declare other peoiple’s views as being without evidence. So, for instance, SWP activist Michael Rosen, a man so divorced from the evidence on how children learn to read that he thinks phonics is “barking at print”, declared in one of his tedious rants directed at Michael Gove that
For you to be able to push through what is fast becoming an exam that will be a major impediment for most young people to develop as learners, you must… ignore all evidence on adolescents and learning.
Similarly, in another Guardian rant, this one so short of actual evidence or argument that Tom Bennett described it as “a joyless donkey ride across the greatest hits of armchair fantasy edu- football”, Suzanne Moore argued that:
Gove, charming as he is, is one of the most profoundly ideological of the lot. One would have thought that a man of his intelligence might push through policies based on evidence. Evidence-based policy-making is all the rage you know. Scientists even do it! But no: the entire education system is now one vast experiment without any aim except the reach of Gove’s ambition.
A third example can be found from this blogger. I was saddened to read:
Like many teachers at a senior level, I have an MA. Three years of hard work in my own time, travelling up to 80 miles on a round trip once a week or so, I wasn’t going to waste my time. The time I used was spent on gathering evidence, from the research of others and from my own work in the classroom. No evidence, no MA basically. There is little evidence in Gove’s ideology.
The one source he has quoted, Daniel Willingham, is a cognitive scientist, not a primary or secondary school teacher, in the USA, a country with perhaps more rigid curriculum rules than our own. He has researched brain mechanisms and memory, and has dismissed the usefulness of learning styles. He appears to be Gove’s guru, and the source of his obsession with rote learning and the rigour of exams.
Phonics: I am not opposed to phonics as such, it is a way of teaching reading, but not the only one. The evidence base was very narrow. A study in Clackmannanshire, the smallest authority in Scotland, is the basis for the introduction of synthetic phonics in England. Too narrow a base, in a part of our nation with a different educational system. The testing too is a political tool. The use of nonsense words, whilst enabling new language learners to show their phonic skills, actually penalised good readers who for example might read the nonsense word ‘dess’ as ‘dress’ because they want to start reading real words to make sense of the nonsense. This appears to have penalised more able readers in their scores, and impacted on schools in the ‘leafier’ suburbs.
So to sum up, the author claims:
- A professor of psychology, who has published two books on education, knows nothing about how learning works.
- The evidence on phonics (Hattie suggested in 2009 that there were 425 available studies of phonics instruction) is reduced to one study that apparently can be ignored due to the size of the local government boundaries.
- The alternative to both the evidence-based discipline of cognitive psychology and the empirical evidence is: the opinion of people who have done MAs in education.
- Probably lots of other things, I just couldn’t bring myself to read any further.
However, if these contributions were not enough to make me wonder if “evidence” is a synonym for “my opinion” and “lack of evidence” is another way of saying “your opinion” there was one blog, widely celebrated on Twitter, that really got my goat. Not because it could compete with the “evidence-based ranting” approach of the above, but because it seemed remarkably plausible until you actually analysed the sources and saw how they had been cherry-picked. This is “The research v the government” from Ian Gilbert. This draws on the Hattie style analysis of education research published by Education Endowment Foundation here.
I have issues with much of the EEF analysis for a few reasons.
1) It looks at effect sizes but seems to ignore Hattie’s claim that when you use this for analysis:
Almost everything works, Ninety per cent of all effect sizes in education are positive. Of the ten per cen that are negative about half are expected (e.g. effects of disruptive students); thus about 95 percent of all things we do have a positive influence on achievement. When teachers claim that they are having a positive effect on achievement or when a policy improves achievement this is almost a trivial claim: virtually everything works. One only needs a pulse and we can improve achievement.
Famously, Hattie’s answer is to compare effect sizes with the average effect size of 0.4. I am a little sceptical about such a cut-off point, but I would suggest that we have every reason to consider effects that are of the order of this “hingepoint” or less to be unproven even when statistically significant.
2) In the absence of decent empirical evidence, the next best thing is the evidence from experimental psychology. To ignore this on the basis of education research, which is usually of a much lower standard, strikes me as a mistake and undermines any claim to be “evidence-based”. There is an exploration of this argument here.
3) The EEF report includes both evidence from studies and opinions which are not clearly drawing upon, and sometimes contradicts, the studies.
Gilbert’s blog ignores these problems, but exploits the third so as to sometime quote from the research conclusions, and sometimes quote from the opinions accompanying them, according to whichever one contradicts the government. For these reasons much of his “evidence” soon seems to be less than convincing when under scrutiny and I will address them each in turn.
Claim 1: Ability grouping harms middle and low attainers.
This is a classic among educational ideologues. I have lost count of the number of times I’ve heard it claimed that this is what all the evidence shows by someone who promptly discovers that they cannot find the evidence in question. While I am yet to do a full review of the evidence myself I can point out that this particular claim is made by the EEF authors on the basis of 4 meta-analyses. 2 of them found a small positive effect for ability grouping. The one that found the largest negative effect (-0.12) for low attainers, according to their own description, actually found a positive effect for homogeneous grouping of +0.12. Given Hattie’s observations about how education research usually finds a positive effect (and a bigger one than this) inverting the result seems unfair. Also, all these meta-analyses are from the 80s and 90s meaning one of the most rigorous bits of education research on ability grouping isn’t included.
Claim 2: There is no evidence for the benefits of school uniform.
This may be an accurate description of the research, but given that elsewhere the opinions of the EEF authors are quoted as evidence it seems a little odd that it has been ignored that they go on to say:
When combined with the development of a school ethos and the improvement of behaviour and discipline, the introduction or enforcement of a school uniform can be successfully included as part of this process.
Claim 3) Performance Pay doesn’t work.
I’m not going to argue with that.
Claim 4) Evidence does not support longer school days.
The EEF authors actually concede there is evidence of effectiveness but doubt whether it is cost effective. It would have been equally possible to quote the following section:
Overall approaches to increasing the length of the school day or the school year add on average two months additional progress to pupils’ attainment over the course of a year. Additionally, research based on international comparisons, looking at average times for schooling in different countries is consistent with this conclusion. However, it should also be noted that pupils from disadvantaged backgrounds benefit by, on average, an additional half a month’s progress relative to their peers suggesting that extending school time can be an effective means to improve learning for pupils who are most at risk of failure.
Here, we have a exact reversal of the way claim 2 was considered. For claim 2, the summary of the evidence was reported but not the opinion of the EEF authors. Here, the opinion of the EEF authors (that it is not cost-effective) is reported but the summary of the evidence (it works) was not reported. Nothing could show more clearly how selective Gilbert is being.
Claim 5) SEAL works.
This is one where the EEF authors are partly to blame. They appear to have quoted a wide variety of studies related to the social and emotional aspects of learning as supporting the effectiveness of SEAL. However, they (unlike Gilbert) admit that when SEAL itself was studied the evidence was not good: “A quasi-experimental evaluation of the impact of the secondary programme did not find a significant impact on attainment in the SEAL schools.”
Claim 6) Nick Gibb was wrong to recognise the success of phonics.
This is again an outrageous selection of opinion over evidence.
The evidence, as the EEF authors admits, indicates “Phonics approaches have been consistently found to be effective in supporting younger readers to master the basics of reading.The approach tends to be more effective than other approaches to early reading (such as whole language or alphabetic approaches)…” Unfortunately the rest of the passage is marred by the usual phonics denialist rhetoric used to obscure the clear message of the evidence with qualifications which can’t actually be deduced from the evidence. Gilbert has quoted only from this obfuscation and opinion.
Claim 7) Despite Gove’s support for sitting in rows, collaborative learning works really well.
This is really one with a lot of background and I intend to blog about it in more detail at a later date. However, it is worth mentioning that the actual research on sitting in rows is ignored here. It is also worth mentioning that the effect size the EEF authors find for collaborative learning is 0.42. Hattie found 0.41. Both are not really distinguishable from Hattie’s “hingepoint” of 0.4 making “collaborative learning” less than clearly effective. This is a case where I would suggest we look at the evidence from psychology. We actually have a 100 years of psychology research supporting “The Ringelmann effect”: a general tendency for people to become less motivated when made to work in groups.
Claim 8) In contrast to the government’s policy of ending ringfencing for one-to-one tuition, such tuition does work.
This is another one where relevant opinions of the EEF authors are ignored. They state that one-to-one tuition is expensive and other alternatives should be considered. Ending ringfencing (as opposed to stopping all one-to-one tuition) actually seems to be in line with this opinion.
Claim 9) Early Years Intervention works, despite a government minister saying Sure Start isn’t a candidate for more money.
Like claim 8, this seems to miss the difference between something being a good use of money and it having an effect. More importantly though, it ignores that a general level of evidence for this form of intervention isn’t necessarily evidence for Sure Start, as the EEF authors acknowledge “Evaluations of Sure Start in the UK do not show consistent positive effects and indicate that some caution is needed when generalising from exceptionally successful examples”
Claim 10) Peer tutoring works and this disproves Gove’’s point of view about collaborative learning.
The immediate problem with this is that Gove’s opposition to collaborative learning was actually an interpretation of a comment about sitting kids in rows (the research on which is again ignored here) so it is far from clear that the evidence on peer tutoring actually has any relevance to what he said. However, even if he does have a general dislike of groupwork, then this cannot be said to disproved by picking the one type of groupwork with a strong positive effect. Why not? I think the following is a really good explanation of why this is not a fair way to do research:
Claim 11: Something about meta-cognition
I don’t even get what is being claimed here.