h1

Don’t let phonics denialists move the goal posts after PIRLS 2016

December 9, 2017

A big difference between scepticism and denialism is that sceptics can identify what evidence would persuade them and then change their position when they have it. Denialists will move the goalposts, acting as if the evidence has no consequences for their arguments. When dealing with denialists you have to constantly remind them of their own arguments otherwise they will simply move on.

The recent PIRLS results, that assessed reading in “4th grade” in 61 countries, and allowed for comparisons between countries and with previous scores was a perfect example of this. This was the first PIRLS cohort to have been through the phonics check. They indicated who was right and who was wrong in their predictions about the effects of the phonics check. The results showed that since the previous round of PIRLS, reading scores in England had improved (to their highest ever) with spectacular gains for the weakest readers.

Supporters of the phonics check had claimed it would improve reading, and that it would be particularly beneficial for the weakest readers. The supporters of the check had made a correct prediction and had reason to feel vindicated. Phonics denialists, who had not predicted the improvement, and often claim that phonics won’t help the weakest readers, immediately started to find reasons these results were irrelevant to the debate. A summary of the denialist arguments can be found here and roughly speaking are:

  1. England is not the only country with an increase in reading scores.
  2. It’s not the only increase ever to have occurred in England.
  3. For the students in the sample, the phonics test score, was not a good predictor of PIRLS scores.
  4. You cannot attribute an improvement to one policy.

Point 3 is just ridiculous. There was a statistically significant correlation of 0.52 on tests 4 years apart where the worst performers would have had extensive remedial help after the first test. The data actually proves the opposite point to the claim being made.

However, the other points are correct. They also show complete amnesia about what phonics denialists were arguing before these results came out.

Did the phonics denialists claim that the phonics check would be followed by improvements in reading but that would prove nothing? Of course not. They suggested reading would be harmed. Opposition politicians, trade unionists, activists and education establishment figures wrote a letter to the Guardian in 2012 claiming:

Many of those working in primary education fear that these tests could undermine rather than benefit children’s progress and development.

Yet somehow reading has improved.

Did phonics denialists claim that an improvement in reading would be irrelevant to assessing the effects of the phonics check before there was evidence of an improvement in reading? Of course not, when Australia was looking at implementing the phonics check a few weeks ago, this was in one of the most popular articles arguing against it:

As the test has been has already been in use for six years in England we are fortunate to be able to learn from their experience. A major evaluation of the test conducted by the Department for Education in England found that the test is not delivering improvements in literacy capabilities, and in fact, is delivering some unwanted side effects…

…What does the research say?

Claim: The phonics test has improved reading results in England since its introduction.

Evidence: Year 1 children in England are certainly getting better at passing the phonics test. Over the past six years, pass rates have increased by 23%. This means around 90% of Year 1 children in England can now successfully read nonsense words like “yune” and “thrand”.

However research has found that the ability to read nonsense words is an unreliable predictor of later reading success.

And so far, the phonics test in England has not improved reading comprehension scores.

So in September, phonics denialists were arguing that a lack of evidence for an improvement in reading showed the phonics test might not be working. Now in December, they argue that an improvement in reading is irrelevant to assessing the effects of the phonics check.

Did phonics denialists argue that PIRLS results were irrelevant to analysis of the policy on phonics before these results came out? Of course not. Janet Downs of the Local Schools Network, a loose online coalition of hard leftists and education progressives, dismissed the 2016 PIRLS results as irrelevant in articles like this one this week. Back in 2012,  she was arguing in an article entitled “10 year-olds from England and Northern Ireland shine like PIRLS in global reading test” that PIRLS results showed that reading was fine without the phonics check and the long tail of underachievement was perfectly normal:

Schools minister, Elizabeth Truss, said the rise in reading performance was “encouraging” but England was being held back because of a “too long tail of under-performance”. While it’s true that 5% of English 10 year-olds didn’t reach the lowest level, 3% of Singapore’s 10 year-olds didn’t reach it either. And in Australia and New Zealand 7% and 8% respectively didn’t reach the lowest level.

Having thrown cold water on these results, Truss used them to promote the controversial phonics test. But PIRLS doesn’t test decoding – it tests comprehension.

So it seems that analysis of PIRLS results was fine when claiming that everything was okay before the phonics check, but cannot be used to judge improvement after the test.

On three counts here, the goal posts have been moved. Of course, there is a limit to what the PIRLS results show. What they show is not that the phonics check definitely caused an improvement in reading, but that the predictions of the supporters of the phonics check were right, and the opponents of the phonics check do not actually respond to evidence. The latter is something we already knew, but now we have even more evidence that opposition to the phonics check is based on ignoring evidence.

Oh, and one final point, one of the claims made by the opponents of the check, and by phonics denialists generally, is that assessing decoding does not tell us anything useful about reading ability in the future. The 2012 letter I quoted above claimed:

…we do not believe that this will help parents know how well their children are learning to read…

They will not show whether a child can understand the words they are reading, nor provide teachers with any information about children’s reading ability they did not already know. The use of made-up words – like snemp, osk, jound – risks confusing children for whom English is a second language and those with special educational needs, and frustrate those who can already read

This argument has already been discredited by looking at Key Stage 1 reading results. However, we now have additional evidence. Here is a graph of average PIRLS score against phonics check scores:

The link between decoding scores and reading performance here is remarkable. So this is another case where, if the phonics denialists were honest, they’d be admitting right now they were wrong about this.

Advertisements
h1

Addendum: A 4th Way phonics denialists will try to fool you

December 2, 2017

Last week I wrote about phonics denialism for the first time in ages. About 24 hours after listing the arguments I most often hear from those who deny the evidence about early reading I remembered I missed one out. So here it is.

4) When phonics doesn’t work

Some children struggle to learn to read. There are a number of possible reasons for this. It could be that the child has some particular difficulty. It could be that even where they were supposedly taught high quality SSP, they weren’t, but learning to read goes wrong sometimes.

Now, in some cases there may be a problem, for instance with hearing, that can be treated directly, but generally most difficulties are lumped together under labels such as dyslexia. This, combined with those children with the opposite aptitude, who learnt to read more quickly than expected, has led to a notion of different types of learners needing different methods of teaching reading. A few decades ago the argument was made that SSP was only suitable for dyslexic students, while everybody else needed the discredited denialist methods. Nowadays the argument has been reversed, and the denialist methods are for those students who make slow progress even with phonics instruction. But neither of these are supported by the evidence, only by the argument that “if evidence based methods didn’t work the first time, we have to resort to magic”. If there were special learners who can only learn to read through denialist methods, nobody has yet found a reliable way to identify them. They still appear largely in anecdotes, not reliable research.

More importantly, what about those with identified reading disabilities, who will be those where phonics may well have failed first time round? Here is what a review of the best evidence for interventions for them found.

“The results revealed that phonics instruction is ….the only approach whose efficacy on reading and spelling performance in children and adolescents with reading disabilities is statistically confirmed..”

Despite the anecdotes, the evidence is that phonics is still the best shot. Some people, however, prefer anecdotes to evidence, so here’s one about how believing that phonics cannot work for a child can be wrong.

This is from Nancy Gedge’s award winning blog in January 2014 about how her son, needed something other than phonics:

The phonics weren’t working, so, …, I made him a set of flashcards of whole words instead.

I was astonished at how quickly he picked it up.  Before long, I had made him a set of interchangeable cards that made sentences, and enjoyed showing him how they fitted together.

I didn’t know that he would go through what felt like a string of teachers who insisted, despite the way his brain processes information more slowly than an ordinary child, that a phonic approach would work, that if they kept on banging his head against the brick wall of unattached letters and sounds he would eventually ‘get’ it.  I wasn’t going to accept any more the battering that his self-esteem was taking at his continual confusion, and my increasing frustration.

I’m not denying that phonics doesn’t work as an excellent way to teach children to read. I use it myself.  But if [he] has taught me anything, it’s that there is always an exception to the rule.

This is what she wrote in June 2014:

If you had asked me three years ago whether [he] would be reading using a synthetic phonic based approach, I would have laughed in your face.  We tried, we really did, but when it came to reading, [he] couldn’t blend efficiently enough to create words out of print, let alone any meaning out of a sentence….

At thirteen years old he has had a year of instruction on a phonics programme, and guess what?  He’s reading.  He’s reading all sorts of things.  From posters to calendars to road traffic signs to Which Caravan Magazine to the blessed reading book that still comes home, after all these years.

Her account (if not her interpretation of the events) fits with the evidence. The best intervention when phonics doesn’t work, is more phonics.

 

 

h1

3 ways phonics denialists will try to fool you

November 25, 2017

I don’t teach reading. The only reason I take an interest in the phonics “debate” is that it’s the one area of teaching where the evidence seems overwhelming. Study after study, review after review (or rather the ones that look at a significant body of empirical evidence) conclude that the closer a method of teaching reading is to Systematic Synthetic Phonics (SSP), the better it is. This is not just the best established empirical result in education, it’s probably the best established result in the entire social sciences. As such, the teaching profession’s willingness to listen to the evidence about this, also indicates our status as evidence-informed, rational professionals.

Unfortunately, like climate change, evolution or vaccination, the conclusions reached are challenging to some ideologies. This means there are those who wish to deny the evidence, usually by confusing people, misleading them or outright lying to them. I wrote about phonics denialism  a few years ago.

Since then, some of the debate has moved on. The introduction of the phonics check has undermined those who claim to be teaching phonics, but not SSP. The check is a test of being able to read the phonetic information in text, if children have been taught phonics successfully they will pass it. Anyone who claims that the check will not work for the kids they have taught phonics to, has not taught phonics, and that seems to have ended that debate. Another, now discredited, argument was that the phonics check would penalise good readers because, despite decades of research indicating the opposite, good readers no longer use phonetic information to read. The results show this isn’t true. So denialists have moved on (or at least they have when there are people around who might challenge them, there are still publishers and newspapers that will print any old nonsense uncritically). Here are the 3 arguments I now hear most often from phonics denialists.

1) The Phonics Fork Ad Hominem

I suppose technically this is 2 arguments, but they are often combined and they are both attacks on the person not the content of their argument. Phonics denialists are most often challenged by one of the following two types of people:

  1. People who are phonics experts.
  2. People who are not phonics experts but know the evidence supports phonics.

The way that the Phonics Fork works is that there is a go to ad hominem argument for both situations.  If they are challenged by somebody who is an expert on phonics, then the phonics denialist will point out that they earn a living from phonics and are, therefore, a vested interest who cannot be trusted. One denialist troll actually used to respond to experts by saying “kerching” – onomatopoeia for the sound of a cash register or a fruit machine paying out – in order to indicate they make money from their expertise and, therefore, cannot be trusted. (Yes, that is the level of sophisticated debate we are dealing with here.) However, if they are challenged by somebody who isn’t in any way an expert, somebody like me, who is only aware of the broad thrust of research and how often denialists have been proven wrong by the evidence, they respond with “well you haven’t taught anyone to read, we shouldn’t listen to you”. This means the only opinions that are permissible in the phonics debate are from those who have been involved in teaching kids to read, but have no expertise in the best way to do it. Which is, of course, the people who are least likely to be in a position to challenge the denialists.

2) Ron Burgundy Syndrome

The consensus amongst the experts about how children learn to read is that once children can decode a word phonetically, then if they understand the word when they speak, then they can understand it when they read it provided they can read fluently enough. If children are not fluent decoders, then they may end up sounding out a word successfully, but not be able to pay attention to meaning at the same time. Also, if they do not know the words in the text they sound out, they will not understand it. Phonics denialists have seized on this as a problem with phonics, rather than a lack of fluency or a lack of vocabulary and claim that non-phonics methods of teaching reading are required to prevent Ron Burgundy Syndrome, an implausible condition where children can decode fluently, reading out familiar words, but having no idea what they’ve said. The only evidence that this condition exists is in the following clip from the film Anchor Man, which I guess for phonics denialists was a documentary not a surreal comedy (warning: contains strong language).

James Murphy wrote a great blogpost listing just some of the evidence that SSP is not just “barking at print” (a common slogan used by denialists) but actually helps understanding too. But phonics denialists will claim that their discredited methods, which undermine good phonics teaching, are necessary if children are to develop “inference skills” or some other ephemera that is meant to underlie comprehension.

3) I’m just saying phonics is not the only part of reading

Perhaps the most common argument I see from phonics denialists these days is one based on equivocation. It is based on phrases such as:

“Phonics is not the only part of reading”.

“There is more to reading than decoding”.

“Reading is more complex than just teaching systematic synthetic phonics”.

All these phrases are wonderfully ambiguous. On the one hand they may be saying children need other things, such as vocabulary and background knowledge, as well as systematic synthetic phonics, to become good readers.

This is something that everybody agrees with. If anybody disagrees with one of the phrases above, a phonics denialist will simply say “well what about vocabulary?” or “well you could sound out words in a language you don’t understand, that wouldn’t be reading” or some other way of arguing (correctly) that phonics alone is not enough without the knowledge needed to understand the language in the text.

However, if not asked to clarify that this is what they mean, phonics denialists will claim that what you need as well as SSP, is teaching using discredited denialist methods: (multi-cuing, word recognition, look and say, etc.) that actually undermine good phonics teaching. It is absolutely vital that, the moment somebody says anything along the lines of “there is more to reading than systematic, synthetic phonics” you pin them down on exactly what they mean. I find just asking “are you advocating multi-cuing?” can be enough to call their bluff.

Another variation on this is to look at the ways a teacher might develop a student’s vocabulary, such as talking to them, using picture books, reading them stories, having interesting books in the classroom, and suggest that teachers who accept the evidence on phonics are against all of these things. In this fantasy, phonics denialists are the only people saving children from 8 hours a day, sat in rows, being drilled in learning letter combinations from a chalkboard while being banned from seeing a book or an illustration.

None of the above 3 denialist tactics are rational arguments. They are tricks used by people who at best intend to confuse, and at worst, intend to deceive. If you see these points being made, I encourage you to challenge them.

 

 

h1

More on academic and non-academic subjects

November 19, 2017

Yesterday I wrote about what I think makes some subjects “academic” and other subjects, while still worthwhile, not academic. The discussion on Twitter immediately afterwards was particularly helpful in helping me reconsider some points and defend others (although by now it largely seems to have been replaced by various progressives arguing against things I never said).

My original argument was that the use of the word “academic” to describe a subject corresponds to those subjects where mastery of the subject was characterised by further study (e.g. history or maths) and not those subjects where mastery is characterised by some distinct activity or skill (e.g. woodwork, painting or football). I acknowledged grey areas (music and MFL can be taught in either academic or non-academic ways) and emphasised that the difference between academic and non-academic subjects does not lead to a value judgement. I also put forward the view that trying to make the non-academic subjects more academic (or vice versa) didn’t do them any favours. I’m still largely happy with what I said but there is something I got wrong, something that I didn’t think about and one new point that I would like to consider.

I will start with the point I got wrong. Because my definition referred to mastery, I think I ended up over-emphasising elite performance. While I still think that the best school plays might indicate the best drama teachers, and the best sports teams might indicate the best games teachers, I should have accepted that general improvement in performance, for everyone at a school, is at least as important as how good the school’s elite are. I should have accepted that participation in, say, sports or the arts might also be important. I will stick to my position that the best drama teaching results in better acting and the best football teaching results in better football playing, but I would not judge these things only by the elite actors and footballers in a school. I stand corrected.

The point I did not think about enough was how subjects are defined and did not make enough effort to be precise in the subjects I talked about. I was amazed that several people expanded subjects way beyond the content I considered them to have. People kept telling me of amazingly academic things that are part of drama that were not acting, from the history of the theatre to the theoretic basis of criticism. I have a GCSE in drama. I did not study one of those things. But, of course, the curriculum changes, particularly in subjects where there has been a deliberate effort to make them seem more academic. I was aware of this in design and technology, and that was why I referred to woodwork and metal work rather than to design and technology. Non-academic subjects are repackaged and have academic content added. Anyone who believes the subjects as they are currently formulated in the GCSE curriculum are definitive will, of course, see them as more academic than they need be. But that is begging the question. I was starting a debate about whether these things are being packaged the right way. We need to look at things from a perspective outside the current framework of assessment and subjects.

To apply my definition, we need to be able to distinguish between the essential and the accidental features of a subject. Acting is essential to learning drama; it is not clear to me that anything else, even if relevant in some ways to drama, is. If the essential elements of a subject are non-academic then it does not matter if the accidental ones are, particularly if they may have been added to the subject to give it more academic credibility. Similarly, learning biology is not essential to learning to play football, and learning how to design a menu is not essential to learning to cook. Perhaps, some subjects will be lacking in essentials and need to be completely rethought and we can perhaps reject any contemporary subjects that have been invented entirely to makes something practical sound more academic. Cookery is a skill in its own right, it shouldn’t have to be repackaged as “home economics” or “food technology”. As far as I can tell some design GCSEs are a way to make some really quite wonderful practical skills look more academic, with coursework folders and written work and without actually testing if somebody can,say, hammer a nail in. PE also raises some issues. I was wrong to think of it as sports. It also covers fitness and we should recognise mastery of it in those who attain a high degree of physical fitness even if they do so without playing sport. Perhaps we would be better off thinking of sport and fitness as two separate subjects. This might seem a contrivance to get round the shortcomings of my definition. However, accepting the current curriculum structures as guidance for subject boundaries and content is not an option, that would simply be accepting decisions that, in some cases, are very recent as telling us the nature of activities that may have been done for thousands of years. We might also get around those subjects that seem to be in grey areas by dividing them into more than one subject, so as to better reflect the nature of the content, rather than the conveniences of the curriculum. Is creative writing really part of the same subject as grammar and literature, or is it an art?

Finally, we have the question of what happens when we go beyond the typical school subjects. There was an assumption among many people that the non-academic subjects I was talking about vocational subjects. Actually, I avoided the word “vocational” as it is not applied consistently in schools. Just because something does not lead to further study, does not mean it is suited to the workplace. A lot of people asked questions that referred to the world outside of schools. Some claimed that if something was studied at university then it must be an academic subject. But of course, universities exist to study things academically. Just because a university might teach sports science, it does not make football an academic subject. You might as well argue that a university teaching criminology makes burglary an academic subject. Universities create new academic disciplines to study things that are not academic disciplines. Sports science, political science, business studies are so called precisely because sports, politics and business are not academic subjects in themselves and have to be made so. The really interesting cases are probably the professions. Are medicine and law academic subjects or not? Perhaps part of the answer here is in the concept of a profession itself. Professions are not just jobs, they are also defined by having a particularly extensive body of knowledge in a way that other jobs do not. Perhaps that is what makes them the hard case, because we struggle to see the dividing line between doing the job and studying that body of knowledge.

Before I finish, I should point out again that this has been an exploration of definitions and the nature of subjects. It has not, and has never been, about policy. Some people think that if you say drama is not an academic subject and it is not best served by being tested in exams, then you would abolish drama GCSE and replace it with nothing and thereby drama would cease to be a priority for schools. I do think drama is more important than drama exams and I really mean this. I would hope getting rid of drama the pseudo-academic subject would not kill drama the art but, if this is a risk, then I am asking here for ways to prevent that, not suggesting it should be allowed to happen. I have no interest in getting rid of non-academic subjects, just replacing pseudo-academic subjects with the actual arts, crafts and sports they currently distort.

h1

Academic and non-academic subjects

November 18, 2017

One of the worst things that happened in education in the 2000s was a seeming reduction in the number of academic subjects. MFL ceased to be compulsory, and some perverse changes in the league tables gave schools an incentive to concentrate on vocational qualifications. In the last few years, particularly with the introduction of the Ebacc and other changes in league table measures, efforts have been made to reverse this. During some of the debates it became clear how divisive it can be to refer to some subjects as “academic” and yet this is something we do quite easily, often without thinking what we mean.

If I had to put into words what I mean when I describe a subject as “academic”, I’d say an academic subject was one where mastery of it was best characterised by further study. The people who are best at history, are historians and they study history; the people who are best at maths are mathematicians, and they study maths, and so on. This immediately creates a distinction between those subjects and some others, where mastery is shown in carrying out a particular activity or skill. We consider the people who are best at football to be footballers. We expect the people who are best at acting to be actors. We consider the people who are best at plumbing to be plumbers. We don’t expect to find these people, who are the best at their subject, to be employed at universities doing research into those subjects. That doesn’t mean you can’t study these things at a university, but the academic study of the subject would be seen as distinct from being the best at the subject, in a way that wouldn’t happen with history or maths.

Once you make this distinction, then you find a few difficult cases. Are the people with the greatest mastery in the field of music those who play music or those who study it? Similarly, who has the greatest mastery of a language? People who speak it, people who write it or people who study it? This leaves some doubt both about how academic MFL is, although probably not classical languages, and also some parts of what is studied in English lessons. In these cases it could be argued either way about whether the subject is academic, or whether parts of it are and parts of it aren’t. Perhaps the best option in those cases is to consider them as subjects that could be taught more or less academically while still being equally focused on some form of mastery of that subject. Whereas in other subjects, we could be more or less academic but there is no dispute as to whether mastery is shown by further study or not. Artists are the best at art; carpenters are the best at woodwork, and nobody would expect a university to be the first place to find them and this contrasts clearly with where we’d find the people who are best at biology and ancient Greek.

Even if you can understand the logic of the distinction I have made, some people are likely to still be furious. The problem is that the place of a subject within schools is often based on how academic it seems. Therefore, even if it seems obvious that football and drama are not the same sort of thing as Latin and physics people will not want to make that distinction. And that’s actually part of the problem here, people will want to make them as much like an academic discipline as possible. They will want budding footballers and actors to have written essays and compiled coursework that has been given a grade, because that’s what academic subjects look like; that’s the route to credibility and legitimacy in those subjects.

It’s also a mistake.

We need to create a culture in schools where the best drama teaching isn’t that which produces the best grades at GCSE or (God forbid) performing arts BTECs, but the one that results in the best actors. A great school production should be seen as a sign of great drama teaching. The school with the best PE teaching is probably not the one where the PE qualifications make the biggest contribution to the league tables, it’s going to be one where their sports teams win and their students have the best chance of becoming professionals in the sports they learnt at school. We need to let arts, sports and crafts be valued in schools as arts, sports and crafts not as pseudo-academic subjects. Being good at football, art or woodwork should not be about getting qualifications, they should be about playing a game, painting or producing a product. Conversely, we should try to stop people making serious academic disciplines into games or entertainment; stop trying to put creativity into maths while taking it out of pottery.

The distinction between academic subjects and non-academic subjects is not a distinction between what is important and what isn’t; it’s a distinction between the ways in which they are important. The arts in particular, are in many ways so much more important than the sciences, that it seems insane to treat them as sciences. I’d love it if the incentives were there so that schools could simultaneously reduce the number of qualifications taken in non-academic subjects, but increase the resources put into those subjects, because the cultural life of a school is as important as exam results. We need to make a distinction between academic subjects and non-academic subjects for the sake of both types of subject.

h1

Why all the research on teacher qualifications is worthless

November 11, 2017

I have a first class degree in pure mathematics. (As I write this I realise that, as a part time teacher in a state school, I am probably one of the lowest paid people in the country with that qualification, worth bearing in mind next time you hear conspiracy theories suggesting the re-emergence of traditionalism in education has been prompted by people attempting to make money out of education.) I have always found my mathematical knowledge and skills an advantage to me in my teaching. Not just when teaching A-level classes, but even when teaching bottom sets it helps to be quick enough to invent my own examples, and to have a good understanding of the importance and structure of mathematics. Like anyone, I’ve seen teachers who had great qualifications but struggled to pass that knowledge on effectively, and those with very unimpressive qualifications who seemed nevertheless to have a gift for explaining, but I assumed these people were exceptions. My personal experience was that, everything else being equal, I was a better maths teacher for having the knowledge of maths which is reflected in my qualifications.

It was a bit of a shock to find that in some schools it was assumed that if you had that sort of qualification then it was assumed that you probably weren’t a good teacher. You were probably only interested in A-level teaching and top sets for GCSE. Your ability to do well in your subject probably meant you’d struggle to understand the difficulties of those who find it challenging. You probably lacked any real skill in the classroom. I took this to be part of the anti-academic culture in many schools and assumed that it might even, in part, be motivated by jealousy of those of us who could have gone into other professions, who did not need to become teachers to earn a middle class income or the opportunity to have a management position. However, I was surprised to learn that the empirical evidence did not support those who believed that, on average, better qualifications made for better teachers. As one educational economist put it on their blog:

“The point is this: there is a general view threading through the teacher recruitment system that applicants with better degrees will make better teachers. I’ll illustrate that in a moment. But all the statistical evidence we have on teacher effectiveness says that that is not true: a teacher’s ability to raise the attainment of her pupils is unrelated to her own academic qualifications.”

I’d seen similar claims elsewhere that, with the possible exception of higher mathematics, teacher qualifications made little difference. This was hard to square with my own experience, but apparently it was what the statistics showed.

Then I heard (from this blogpost) about Berkson’s Paradox, a statistical anomaly that can explain a number of counter-intuitive results. This is a selection bias where we are looking at the connection between two events (like being an effective teacher and having a good qualification) and we look only at data where at least one of the events we are interested in happens*. The blogpost above uses this diagram to show what happens:

In the first picture, data for the whole population is shown. There is a positive correlation between two variables. In the second picture, those data points where both variables are low have been removed, and this selection reverses the direction of the correlation. This can happen in a number of situations. (The first 3 examples are from the blogpost mentioned above).

1) Studies of intellectual ability and academic motivation among college students. We might expect these two to be correlated, but if we only look at students who successfully made it into good colleges or universities, then those who are lacking in both ability and motivation will probably be excluded from the sample. Therefore, you may find a negative correlation between motivation and ability.

2) The correlation between research productivity and teaching effectiveness. We would not expect good researchers in universities to be bad teachers. But, a university would have no reason to employ a bad researcher who was a bad teacher, so again the sample will be altered. Therefore, you may find a negative correlation between research productivity and teaching effectiveness in academics.

3) The burger-fry tradeoff. We would not expect restaurants that are good at cooking burgers to be bad at cooking chips. If anything, we might expert culinary brilliance to transfer from cooking one item to another. But, if you like burgers and you like chips, you have no reason to go to a restaurant that is bad at both, so the restaurants you go to are a biased sample. Therefore you might find a negative correlation between quality of burgers and tastiness of chips.

4) “Why are handsome men such jerks?” – Ellenberg’s Paradox There’s no reason to assume good looking people have terrible personalities. But if you don’t date people who are ugly and have terrible personalities, then your sample of dating opportunities, will be biased. Therefore, you may find that in your experience, beautiful people are more likely to be tedious or unpleasant.

There’s every reason to think this might affect studies of whether better qualifications lead to better teaching. Imagine if my subjective impression is right, and better qualifications mean better subject knowledge, which means better teaching. If we looked at a sample of potential teachers, we might expect a correlation like this between teacher qualifications and a combined measure of the other attributes that make for good teaching.

Now, if we restrict ourselves only to people who are employed by schools, the sample changes. No school has good reason to employ people who are lacking in qualifications and in all other attributes that might aid good teaching. We could expect a sample looking at actual teachers to have fewer points in the bottom left corner. So removing those points from our original picture, it might now look like this.

And our correlation has gone. Better qualifications no longer predict other attributes. It could even be worse than this. If the elite schools are likely to get more and better qualified applicants, and employ the best of those who are well-qualified, then this might even remove teachers in the top right corner. Our average school could even end up with a correlation like this.

And suddenly we have a situation where good qualifications are negatively correlated with other attributes of being a good teacher. Yet, this is all from a situation where we started from assuming that good qualifications help teaching.

Now please don’t take this too seriously. Don’t start making teachers fail their performance management for being over qualified. All the above graphs are invented to illustrate a point, I am not seriously claiming that those of us with better qualifications are, if we are in average schools, worse teachers than the less qualified (although suddenly those exceptions I mentioned earlier look less likely to be exceptions). What I am pointing out is that if we look at teachers in schools and look for a correlation between qualifications and teaching effectiveness we are likely to find no correlation or even a negative correlation, even if better qualifications do make us better teachers. The research on teacher effectiveness which concludes qualifications don’t help, actually tells us nothing about the effects, system wide, of better qualified teachers. The same would probably go for other measures of teacher knowledge or assessment of any teacher attribute that might be important enough to affect one’s chances of getting a job as a teacher. All research comparing desirable teacher qualities (or at least those teacher qualities significantly affecting the chance of being employed) and teacher effectiveness which is based on looking at samples of actual teachers (rather than deliberate experiments) is likely to be worthless. My intuitions may well be mistaken, but the research doesn’t actually give me any reason to throw them out.

 

*I think technically Berkson’s Paradox might only apply to the case where selection turns a situation where there is no correlation to one where there is, but in this post I will treat it as essentially the same as where selection removes or reverses a correlation.

h1

Let’s not argue over why we can’t make a difference

November 4, 2017

Despite all the talk among progressives of needing new types of education, one of the dividing lines in contemporary debate tends to be about the potential for massive improvements in education. And I do mean “tends” here. This dividing line is a blurry one and there are many exceptions. However, traditionalists often see our education system as failing, and even if we don’t use the language of failure, we do usually argue that massive improvements can be made. We are usually confident that evidence based teaching of early reading can cause a large change in literacy. We do tend to argue that if more knowledge was taught, taught better and taught in a way that would be retained, academic standards could be transformed. We have often explained how better discipline in schools can be the difference between kids getting an education, and many kids being in school, but not actually learning. Meanwhile, progressives are more likely to argue that this line, even when it comes from teachers, is “teacher bashing” and if politicians would just leave the education system alone, provide lots of money, make society fair and equal, and let educationalists and managers get on with their job unhindered by accountability or change, everything would be just fine, or at the very least, the problems will not be those caused by schools.

This has been the background to interpretation of research that shows that, on average, schools don’t make that much difference to outcomes. There are two lines of research that show this. One type shows huge correlations between social class and educational outcomes. Another type of research shows, on average, outcomes can be predicted based on one’s genetic inheritance, either by showing that who you share genes with matters more than who raises you, or by attempting to measure “innate” rather than learned abilities. Both types of research has been used to show schools make little difference on average. Both types of research have been used to justify similar ideas. If differences between students matter more than differences between schools, people argue:

  1. there can be no type of education or knowledge suitable for all;
  2. that the levels of educational failure we see, are, from a school’s point of view inevitable and schools cannot be held responsible for them;
  3. that there is no way to improve our schools other than making them better at catering to student difference.

Roughly speaking, these positions can be described as “determinist”. In my experience, modern progressives are more likely to be social determinists. They are often on the political left and believe that as social class determines outcomes, the important thing for schools to do is to lead the oppressed to political maturity. However, historically, many progressives were genetic determinists. Before the second world war, eugenics was seen as a progressive cause,  accepted by enlightened leftists like the Fabian Society, and opposed by regressive conservatives (like the Catholic writer G.K. Chesterton). There is an excellent chapter in Left Back: A Century of Battles over School Reform about the influence ideas of measuring innate ability had on some progressive educationalists.

As you can imagine, the history here is pretty ugly and there are few advocates of progressive education now who justify their position in terms of genetics, although talk of naturally “academic” and “non-academic” kids is still part of progressive discourse. The nearest you can find now to somebody using genetics to justify progressive education would probably be behaviourial geneticist Robert Plomin, who is quoted in the Guardian as concluding the following from his work on how genes determine outcomes:

“Education is still focused on a one-size-fits-all approach and if genetics tells us anything it’s that children are different in how easily they learn and what they like to learn. Forcing them into this one academic approach is going to make some children confront failure a lot and it doesn’t seem a wise approach. It ought to be more personalised,”

The arguments over which is the right sort of determinism have become vicious. Some now consider the idea that genes influence outcomes, or that we might measure or study innate ability, to be utterly abhorrent and only to be mentioned by racists and eugenicists. Sometimes, even those of us who merely challenge social determinism, without advocating genetic determinism, are tarred with the same brush. The situation is not helped by the fact that the empirical evidence seems to favour the genetic determinists, but there are many problems with those empirical methods (not the least of which is that correlation is not causation) and even more problems with drawing out practical lessons from that evidence. This means that, a lot of the time, people are arguing over what the empirical evidence actually means, whether it is ethical to consider empirical results in the first place and, inevitably, whether the views they don’t like should ever be expressed.

For those educational traditionalists who believe that much can be done to improve schools, the issue is educationally irrelevant. It does not matter what, on average, causes educational outcomes if, on average, outcomes aren’t good enough. If we believe schools can make a huge difference to education, then arguing over what determines outcomes when schools aren’t making a huge difference, is a waste of time. Imagine if people had decided there was no point inventing a polio vaccine until we knew whether the effects of polio were best predicted by social class or genetic make up. To us, it’s like Liliputians and Belifuscudians arguing over which end of a boiled egg to crack. The important thing is not whether we are writing kids off because of their genes or because of their social class, but whether we are writing off kids at all. I’ll weigh into the debate if I see people arguing for censorship or doing their maths incorrectly, but I don’t really care about the core issue except in how it affects how we end up treating people. I’ll happily oppose both eugenics and totalitarian social engineering. I have no interest in choosing between Brave New World and 1984.

%d bloggers like this: