h1

I told you so: Evidence and the Chartered College of Teaching

April 21, 2018

I wasn’t planning to write about the Chartered College Of Teaching again. Nobody involved seems to care about my criticisms, so I’m sure that when I write about it the only effect is that I publicise them and probably get them a few more members.

But no blogger can resist the chance to say “I told you so”, so I have to comment on the bizarre saga of Greg Ashman’s article on metacognition which has been all over his blog and Twitter lately.

Back in 2014 I wrote about plans for a new professional body for teachers. I discussed at length what it would take for it to be something other than a new version of the despised GTC(E) and the potential problems if it tried to represent too many interest groups, or particular ideologies rather than the profession. Some supporters of the plans suggested that if the College focused on disseminating research and evidence then that could avoid ensure that it wasn’t seen as a partisan interest group. I wrote this blogpost explaining how debates around the use of evidence were actually highly contentious and partisan. I gave examples of views of other people involved in education about education research and concluded:

Now if you know anything about my views, and what I consider to be the evidence that underpins them, I find it impossible to imagine that my disagreements with any of the above can be resolved by reference to evidence. I am not arguing here that I cannot be part of a College of Teaching which includes people with views like those above, but I am certain that no amount of evidence or research is going to allow us all to support a single College of Teaching that claims to be promoting “what the research shows”. Research and evidence are divisive, not unifying, forces in education.

The last few weeks have served to illustrate this. The Chartered College of Teaching has a publication called Impact. According to their website:

 [Impact] supports the teaching community by promoting discussion around evidence in the classroom, and enabling teachers to share and reflect on their use of research.

Some great people are involved with it, and although I haven’t read it, I’ve heard wonderful things about the first issue. The second issue was to be edited by Jonathon Sharples of the EEF (the organisation I blogged about here). Among the topics they requested articles on was “Metacognition, self-regulation” which is one of those broad educational ideas (see “thinking skills”, “oracy” and “creativity” for other examples) which people build all sorts of teaching ideas around, without any teacher ever being clear precisely what it covers. The EEF has been promoting metacognition, for reasons that are somewhat mystifying, for a while now. Teacher, Greg Ashman, pointed out the problems with the EEF’s allegiance to this idea in a blogpost in January entitled Is ‘metacognition and self-regulation’ an actual thing? 

Greg, without hiding his cynicism, suggested that this might end up being the main focus of the issue of Impact and was told “submit an abstract”.

He did so, and the abstract made clear that his view was:

… the category of meta-cognition and self-regulation seems to have been stitched together from a range of different beasts, much like the mythical chimera… Practitioners should therefore be wary of any simplistic claims made for this category of intervention…

(The abstract can be found here). The abstract was accepted, and he was asked to write the article. He wrote it. It then went to peer review. It is at this point things got a little odd. One reviewer, Dylan Wiliam, claims to have said:

“The article is provocative, but essentially well-argued, and worth including as a prompt for debate. It would be great if someone from EEF could respond in a subsequent issue, because then it would mark out Impact as a forum for debate.”

The first sentence of this was sent to Greg. Another reviewer, in another brief comment claimed Greg’s argument was not clear. Another of the three peer reviewers, however, sent two pages largely arguing against Greg’s position on the grounds that a review of the wider literature would find that the approach the EEF had used was well-established, and therefore Greg was wrong to think that what the EEF had done was “astonishing”.

Reading these three reviews (you can find them with 4 other reviews here) you can’t hope but noticing that a lot of the problems are about a lack of clarity about what Impact is for. One reviewer recommended Greg’s article as a provocation for debate. One was scathing that it did not address the wider ideas of education researchers, but addressed only something that had been aimed at teachers. I can see both points of view, it all comes down to whether Impact is for teachers to debate ideas that affect them, or for education researchers to discuss research. I think this reflects the lack of clarity about what the Chartered College of Teaching is for; is it for teachers or for educationalists? Additionally, reviewers do not seem to have been clearly asked whether the article should be rejected, amended or accepted. Had it done so, it would have been 2 to 1 against, and perfectly legitimate to reject it outright.

Instead, Greg was asked to amend his article. He did so. It was then sent out to 4 more reviewers. I don’t know why. Worse, these additional reviewers did not seem to address the issues raised by the first reviewers, but commented on the tone of the piece and raised new issues about accuracy, which Greg didn’t have time to respond to (although he is now of the view that none of the points about accuracy were correct).

As a result, Greg could not address the further peer reviews and the article could not be published. A confused peer review process, and apparent confusion about the purpose of the journal, had served to exclude an interesting article, and a perspective relevant to teachers, from the journal, although not from the website. Greg, who I think had been sceptical from the beginning about whether the journal would ever accept his work, described on Twitter and in blogs what had happened.

Then two further odd things happened.

Firstly, supporters of the College began criticising Greg. It was assumed that he was bitter about rejection, rather than concerned about the process (which seemed to have wasted his time). People implied that Greg was so desperate to be published, that he was a bad loser seeking revenge for a personal blow to himself and his credibility. Given that Greg’s latest book is available for pre-order here; given the number of other people willing to publish Greg’s views, and given the praise from Dylan Wiliam and others for that article, such a line of attack seems implausible as well as unpleasant.

Secondly, the Chartered College of Teaching Twitter account commented on the matter in a long thread. In the thread they falsely claimed “At no point did reviewers take issue with the opinions”.

Since then, Greg has released the peer reviews and proved that this was false, and been attacked for that. Numerous supporters of the College have argued that a publicly funded professional body making a false statement about a teacher is not as unethical as a teacher releasing the evidence that the statement is false.  More bizarrely, others have, apparently sincerely, claimed that the arguments the reviewer made against Greg’s views was not “taking issue with his opinons”.

Yes, really. Highlights of that discussion include people claiming that Greg’s opinions were not opinions but assertions, conclusions or claims and that arguing against his opinions was not taking issue with them, but challenging them, objecting to them or discussing the words used to express them. At times, those who defended the false statement could not even remember which bit of sophistry they were currently using:

These tweets, highlighted by Greg, are by a professor of education

I’m left amazed at the cult-like devotion to the Chartered College Of Teaching that exists among a small minority (many of whom aren’t teachers) who are willing to make themselves look silly rather than admit the College’s mistakes. I’m left appalled that the College has still not apologised to Greg for the false statement. I’m left grateful to Michael Fordham for this Twitter thread discussing the rights and wrongs of peer review in professional publications. But mainly I’m left smugly saying “I told you so”. Interpretations of evidence cannot unite the teaching profession. A professional association for teachers needs something else to underpin it other than research. I would suggest a belief in teacher professionalism would be a far better basis for building a professional association for teachers.

Advertisements

23 comments

  1. I know I’ve been part of this messy discussion Andrew, and both challenged the neutrality of your position and queried the overall uniqueness of Greg grumbling about not having his article published.

    Two things I’d like to say:

    Yes – I fully see now that Greg’s experience was definitely worth debating in detail. This wasn’t a good experience for him, and nor did it show a coherent – and easy to let pass – process for the compilation of the Impact magazine (or at least that issue). I will indeed indulge you with your “I told you so” moment!😁 You did indeed foresee such complications….

    The second focus though is about what we should take from this… Clarity about the purpose of the Impact magazine is important (it isn’t the second issue by the way). I personally definitely see it as a digest which can be left on staff-room tables to give some thoughtful – but useful – clarity regarding what things everyday teachers should best spend their time focusing on. I don’t see it as being for educationalists. In that respect, there are two challenges: One is getting a ‘mode’ position right (always debatable) regarding what can be confidently declared as reliable advice and what is thoughtfully up for discussion.

    The second challenge is how the decisions about such inclusion get made. How does the college go about appointing people to have editorial overview of such decisions…?

    My thoughts are two-fold: It is probably much better to have the position of the editor up-front – to allow the prejudices of the edition to be evident. Then the challenge is simply to ensure that this editorial viewpoint is varied over time.

    Secondly though, even if leaderdhip and membership of the college was restricted to serving teachers, these same issues would arise… We all have different values and perspectives when it comes to choosing evidence, and narrowing the college to a particular interest group – as well as reducing agendas – also reduces the range of experience and hence perspective permitted in the discussion.

    I think we need to simply persist in pushing through with what we’ve got – and supportively nudging forward it’s evolution – or forget it completely.


  2. No surprise there then…. for me it just reinforces the impression that the teaching profession is incapable of rising above endless squabbling and politicking within itself. To be fair, that is in part down to the elusive nature of what teachers do, but it is not helped by those who insist on amplifying it.

    I derive most comfort from knowing that the vast majority of teachers I worked with didn’t give a stuff about all the abstract discussion – they were far too busy actually trying to educate their pupils. That may seem ignorant as a position, but I increasingly think it is the only sensible/bearable one.


    • true, but they don’t realise this so-called evidence effects their daily lives as it directs educational policy and that of their students too!!!

      It’s like saying most Americans don’t care about the debate between Donald Trump and James Comey, but at the debate lies the very issue of the problems.


  3. Reblogged this on The Echo Chamber.


  4. “Research and evidence are divisive, not unifying, forces in education.” What an extraordinary statement. Is it true?

    Compare with in medicine. There’s always a few anti-research/evidence nuts; anti-vaxxers and homeopaths etc. But they’re very marginal.

    Research and evidence unites pretty much every doctor you’ll ever come across in the NHS, for example.


      • I did. I think you, and the people you cite, understandably, don’t realise how good research could unite teachers, because education is like medicine was 100 years ago, before the first randomised trials. There have been no high quality trials of any important intervention with follow-up to show an effect on “what matters”. So people don’t realise you can separate “what works” from “what matters”. Parents, or sometimes pupils or politicians, choose what matters. But with good research we can agree how to get there. The reason you have so little faith in education research is that right now it is 99.999% utterly awful, ideological, pretentious rubbish. Sorry. Forgive me. :)


        • I think you are being optimistic. If education is like medicine, it’s more like the medicine of 1017 than 1917. It’s not the case that people are committed to the scientific study of learning, but not to an appropriate methodology for judging interventions. It’s the case that “expert opinion” is seen by many as the only valid evidence.


        • I think the ‘little faith’ also has a lot to do with building a culture where you can dismiss people, rather than engage with arguments.


        • One important issue about educational research which this event shows, is the outdated value of closed peer review and proprietary publication.

          Why is the journal ‘Impact’ not published open access?
          Why isn’t there an educational research print server (https://en.wikipedia.org/wiki/Preprint) suitable for teachers and researchers?
          Why didn’t Mr Ashman choose either an open access journal or a print server such as ‘socArxiv’? This would avoid petty “gate-keepers”?


  5. The response of the Chartered College is to be welcomed. It’s giving Greg far more publicity than his article would have otherwise obtained, and raised some serious question about the College’s entitlement to public funds. It has also revealed some appalling errors over the question of (specifically) metacognition and self-regulation, and more generally the way disparate studies are conflated by researchers to create seriously misleading effect sizes. It will also serve to put a question mark over EEF data and their right to public funding. Certain, Greg has caught them with their trousers down–take this comment:

    “And I have been particularly critical of the ‘metacognition and self-regulation’ strand of the toolkit which, as I explained in the recent article that the UK’s Chartered College of Teaching declined to print, appears to be a chimera; a monster stitched together from quite disparate things. What does a writing intervention where students are explicitly taught how to plan, draft, edit and revise their writing – also known as ‘teaching English’ as Joe Nutt drily observed on Twitter – have in common with ‘philosophy’ lessons where children discuss whether it is okay to hit a teddy bear? Not much. It’s astonishing that such wildly different interventions would be classed as the same thing. Medical researchers, on whose model of meta-analysis the EEF has built, would never group together such diverse approaches, with such different aims, methods and outcomes, and try to compute an effect size as if they are all the same thing.”


  6. I agree with Greg’s point that metacognition is too broadly defined to be useful, I agree on the need for more open debate in education, and I believe that education is sufficiently charged with ideological prejudice that the charge of no-platforming must always be taken taken seriously. But it seems to me that Greg is being a little churlish. The editor of Impact is right, in my view, to say that the tone of the piece is too hyperbolic and the language needs to be more measured. Greg weakens his position by throwing the toys out of the pram – he should have worked with the editor to modify his language and continued to argue his case on accuracy.

    I agree with Jim Thornton on research. It is true that evidence can be cited to support anything – but then it becomes fake evidence. Much of the evidence in education is of this kind. The fact that we do research in education so badly is not a reason why we should not aspire to do it well.

    Objective research, answering the question “what works?” is, I accept, dependent on deciding what it is you want to do – and you might say that this second question is subjective. Yet it is not a question for teachers to determine. The education service is set up by society at large. It is for society, acting through some sort of wider, politically medicated conversation, to determine the objectives of education (i.e. the curriculum) and for teachers to develop their expertise in how to meet those objectives (i.e. pedagogy) or, if they are wholly out of sympathy of the educational objectives that they are being asked to pursue, to leave the profession.


    • Where did the editor say any of those things?


      • You are referring to my comments about tone? Greg says “I was surprised to read comments objecting to my tone” and that “I was given a couple of days to revise my article for tone and accuracy” but that “I declined, partly due to the fact that I didn’t see the need”. I disagree – the tone was too hyperbolic for a formal journal. He should have modified the tone and continued to argue the case on accuracy (there is always a need to argue your case).


        • You seem to be confusing the editors and the peer reviewers. Absolutely fine for editors to comment on style and tone. Odd for peer reviewers to do it. Stranger still to be given extra peer reviews at the last minute asking for changes in tone.


          • So your objection is not that Greg was asked to modify his tone, but that the issue of tone was raised by reviewers and not just by the editor? Do you object that Dylan Wiliam also commented on tone? You are right – I don’t see the need to make a big deal about who made the request. For me, that is part of Impact’s internal process and is their business, not ours.


    • Except teachers, like Greg, aren’t welcomed when they enter discussions on pedagogy Crispin. He tried, they rejected him.

      Nor are parents much welcomed in discussions on the aims of education, not social aspects such as exclusion (we know they would give the “wrong” answers on discipline).

      A lot of academics want to control curriculum and pedagogy.


      • Chester, I am not saying you aren’t right. I agree with Gregs point about the definition of metacognition (see my comment at https://gregashman.wordpress.com/2018/04/17/the-article-that-englands-chartered-college-will-not-print/#comment-12713). What I am objecting to is the method of pursuing the debate. These things need to be pursued meticulously, following the argument, not by name-calling. Andrew gave a talk about methods of argument in one of the early ResearchEd conferences and one of the points he made was the need for charity towards your opponents – i.e. placing the most generous interpretation that one reasonably can on their motives and making as much sense as you can of their arguments. Its not because you want to be nice to them – its because you want to run the argument down. Imputing unworthy motives to people is a cop-out from the real business at hand.


        • On the question of metacognition and self-regulation, only two conclusions are possible: either the EEF was being driven by ideology, or they were careless to a degree that would serve to question their right to public funds. I’d judge that Greg was being quite restrained in his comments, and for the CCT to complain about his tone is just a wee bit rich.


          • Unless you admit a third possibility, that the other guy was right all along, you abandon all hope of civilized, evidence-led debate. Which seems to be what has happened here.


  7. The whole thing is just bizarre.
    I renewed for another year but I am still a little sceptical about the whole thing. There is certainly a fervent core of ed researchers and consultants who are championing the cause, many of my ‘mum’ friends who are primary teachers don’t even know CCT exists.


  8. […] Teaching in British schools « I told you so: Evidence and the Chartered College of Teaching […]


  9. […] it has badly missed its membership targets and the peer review system for its journal turned into a farce. I would have thought these were the sort of things that outside experts would get right, even if […]



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: