h1

Why all the research on teacher qualifications is worthless

November 11, 2017

I have a first class degree in pure mathematics. (As I write this I realise that, as a part time teacher in a state school, I am probably one of the lowest paid people in the country with that qualification, worth bearing in mind next time you hear conspiracy theories suggesting the re-emergence of traditionalism in education has been prompted by people attempting to make money out of education.) I have always found my mathematical knowledge and skills an advantage to me in my teaching. Not just when teaching A-level classes, but even when teaching bottom sets it helps to be quick enough to invent my own examples, and to have a good understanding of the importance and structure of mathematics. Like anyone, I’ve seen teachers who had great qualifications but struggled to pass that knowledge on effectively, and those with very unimpressive qualifications who seemed nevertheless to have a gift for explaining, but I assumed these people were exceptions. My personal experience was that, everything else being equal, I was a better maths teacher for having the knowledge of maths which is reflected in my qualifications.

It was a bit of a shock to find that in some schools it was assumed that if you had that sort of qualification then it was assumed that you probably weren’t a good teacher. You were probably only interested in A-level teaching and top sets for GCSE. Your ability to do well in your subject probably meant you’d struggle to understand the difficulties of those who find it challenging. You probably lacked any real skill in the classroom. I took this to be part of the anti-academic culture in many schools and assumed that it might even, in part, be motivated by jealousy of those of us who could have gone into other professions, who did not need to become teachers to earn a middle class income or the opportunity to have a management position. However, I was surprised to learn that the empirical evidence did not support those who believed that, on average, better qualifications made for better teachers. As one educational economist put it on their blog:

“The point is this: there is a general view threading through the teacher recruitment system that applicants with better degrees will make better teachers. I’ll illustrate that in a moment. But all the statistical evidence we have on teacher effectiveness says that that is not true: a teacher’s ability to raise the attainment of her pupils is unrelated to her own academic qualifications.”

I’d seen similar claims elsewhere that, with the possible exception of higher mathematics, teacher qualifications made little difference. This was hard to square with my own experience, but apparently it was what the statistics showed.

Then I heard (from this blogpost) about Berkson’s Paradox, a statistical anomaly that can explain a number of counter-intuitive results. This is a selection bias where we are looking at the connection between two events (like being an effective teacher and having a good qualification) and we look only at data where at least one of the events we are interested in happens*. The blogpost above uses this diagram to show what happens:

In the first picture, data for the whole population is shown. There is a positive correlation between two variables. In the second picture, those data points where both variables are low have been removed, and this selection reverses the direction of the correlation. This can happen in a number of situations. (The first 3 examples are from the blogpost mentioned above).

1) Studies of intellectual ability and academic motivation among college students. We might expect these two to be correlated, but if we only look at students who successfully made it into good colleges or universities, then those who are lacking in both ability and motivation will probably be excluded from the sample. Therefore, you may find a negative correlation between motivation and ability.

2) The correlation between research productivity and teaching effectiveness. We would not expect good researchers in universities to be bad teachers. But, a university would have no reason to employ a bad researcher who was a bad teacher, so again the sample will be altered. Therefore, you may find a negative correlation between research productivity and teaching effectiveness in academics.

3) The burger-fry tradeoff. We would not expect restaurants that are good at cooking burgers to be bad at cooking chips. If anything, we might expert culinary brilliance to transfer from cooking one item to another. But, if you like burgers and you like chips, you have no reason to go to a restaurant that is bad at both, so the restaurants you go to are a biased sample. Therefore you might find a negative correlation between quality of burgers and tastiness of chips.

4) “Why are handsome men such jerks?” – Ellenberg’s Paradox There’s no reason to assume good looking people have terrible personalities. But if you don’t date people who are ugly and have terrible personalities, then your sample of dating opportunities, will be biased. Therefore, you may find that in your experience, beautiful people are more likely to be tedious or unpleasant.

There’s every reason to think this might affect studies of whether better qualifications lead to better teaching. Imagine if my subjective impression is right, and better qualifications mean better subject knowledge, which means better teaching. If we looked at a sample of potential teachers, we might expect a correlation like this between teacher qualifications and a combined measure of the other attributes that make for good teaching.

Now, if we restrict ourselves only to people who are employed by schools, the sample changes. No school has good reason to employ people who are lacking in qualifications and in all other attributes that might aid good teaching. We could expect a sample looking at actual teachers to have fewer points in the bottom left corner. So removing those points from our original picture, it might now look like this.

And our correlation has gone. Better qualifications no longer predict other attributes. It could even be worse than this. If the elite schools are likely to get more and better qualified applicants, and employ the best of those who are well-qualified, then this might even remove teachers in the top right corner. Our average school could even end up with a correlation like this.

And suddenly we have a situation where good qualifications are negatively correlated with other attributes of being a good teacher. Yet, this is all from a situation where we started from assuming that good qualifications help teaching.

Now please don’t take this too seriously. Don’t start making teachers fail their performance management for being over qualified. All the above graphs are invented to illustrate a point, I am not seriously claiming that those of us with better qualifications are, if we are in average schools, worse teachers than the less qualified (although suddenly those exceptions I mentioned earlier look less likely to be exceptions). What I am pointing out is that if we look at teachers in schools and look for a correlation between qualifications and teaching effectiveness we are likely to find no correlation or even a negative correlation, even if better qualifications do make us better teachers. The research on teacher effectiveness which concludes qualifications don’t help, actually tells us nothing about the effects, system wide, of better qualified teachers. The same would probably go for other measures of teacher knowledge or assessment of any teacher attribute that might be important enough to affect one’s chances of getting a job as a teacher. All research comparing desirable teacher qualities (or at least those teacher qualities significantly affecting the chance of being employed) and teacher effectiveness which is based on looking at samples of actual teachers (rather than deliberate experiments) is likely to be worthless. My intuitions may well be mistaken, but the research doesn’t actually give me any reason to throw them out.

 

*I think technically Berkson’s Paradox might only apply to the case where selection turns a situation where there is no correlation to one where there is, but in this post I will treat it as essentially the same as where selection removes or reverses a correlation.

Advertisements

15 comments

  1. ‘Now, if we restrict ourselves only to people who are employed by schools, the sample changes.’

    But isn’t that the sample we should be working with?

    Schools don’t employ people with no qualifications who are rotten teachers.

    Including those people in our sample will skew the results in ways we don’t want.

    Is what I have written correct? I don’t actually know.


    • Depends. If you are a politician trying to decide whether teacher training should target recruitment of the well qualified, or a school trying to judge a new teacher from their qualifications, then those currently working in schools as teachers are not a good sample to base your conclusions on.


  2. Reblogged this on DT & Engineering Teaching Resources and commented:
    Why all the research on teacher qualifications is worthless


  3. Really well written interesting article. Probably need to read again to fully take in! Suspect your intuition is sound with mathematics helping to link/inform your ideas. Think in depth study with yourself and a few others could give more insights. Interesting area of inquiry – well worth pursuing – could connect with emotions and learning. Hope you pursue and write more about! Good luck for the future!


  4. It would be worth comparing residuals to the fit. If first were normal, the second won’t be. I’m tempted to do this tonight and show. Perhaps speculate on a “measure” to detect. (I wouldn’t be surprised that if I do something, it will turn out its been done and a measure exists.)


  5. This is an interesting idea. Someone with a strong background in quantitative analysis of education datasets might be able to tell you whether this is something that has been considered thoroughly already, or not.
    I’ve had about fifteen different thoughts, some of which are circular, but I’ve come down to an issue about how your initial graph is set up. In your arguments you’ve said that it’s based on your subjective impression that “better qualifications mean better subject knowledge, which means better teaching” but your graph shows a correlation between qualifications and other attributes. That’s not the same thing. Should your graph not show a correlation directly between qualifications and teaching effectiveness? And then, would you not expect (hope?) that the process of selection, by both ITT, employers, and elite schools, would at least be trying to make horizontal slices in the sample, rather than diagonal ones? I’m not sure.


    • Fair enough. Trying to relate it more directly to the selection process.


  6. “Now please don’t take this too seriously. ”

    I won’t. Mainly because your example is all made up.

    “Imagine if…”
    “We might expect…”
    “We could expect…”


    • I’m sorry to hear that you are unable to learn from considering a hypothetical example, no matter how plausible.


      • There you go, throwing assumptions all over the place. I understand the arguement and I’m quite interested in Berkson’s Paradox, just not the unrealistic way you shoehorn it into teacher effectiveness.
        Hence not taking your argument too seriously, as requested.

        I probably score low on “other attributes”


        • I’m not assuming anything. You dismissed a hypothetical example on the grounds it was invented. Do you now admit that was a silly point?


          • Reading the actual paradox, the two events need to be independent. This is a pretty vital point missing from your argument.


  7. EDIT: I suppose the independence is implied in your footnote, although independence and non-correlation are not quite the same.


  8. I don’t think teaching skill is likely to be well correlated with qualifications.
    It will more likely correlate with academic ability and hard work. I would employ a BSc with A grades over an MSc with C grades, all else being equal. I would employ a Philosophy degree with Bs over an English degree with Cs too, even to teach English.
    I never ever use my university knowledge past 2nd year. Quantum effects aren’t in school chemistry. Partial derivatives aren’t in my local curriculum. I find I do read generalist books and websites to broaden my knowledge, but I never crack open my old textbooks, as they are way too specialised.
    In the past qualifications were a good proxy for ability and work ethic. Now every second person has a degree, I doubt it is half so good.
    So any study that uses qualifications as a proxy for ability is pretty useless in my opinion. Perhaps people with good degrees might be going to industry leaving the highly qualified but dimmer brethren looking at teaching as a way to use their degrees.
    I am up for studies where the quality of the degree up to third year is compared. But no extra value for second degrees and Masters etc.


  9. Some Ontario Faculty of Education place no more than 50% of your acceptance weighting on grades. The rest is determined by what evidence you can provide that you’re truly committed to teaching/helping others. Best yet, your application can NOT be more than what the application form allows for. Now add to this an interview as some schools do and you’ve got a pretty good chance that you’ll get good hires.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: