This, from Matthew Hunter’s blog, says it all, really

goodbyemisterhunter

The following post is an email that I received last week from a fellow teacher. The content will be familiar to many: pressure to adopt child-centred teaching methods; disillusionment with PGCE course; atrocious behaviour in schools; excuses making from school senior management. This email sews it all together in a particularly moving way. It is distressing to think how many talented and enthusiastic teachers have been driven out of the profession, or into the independent sector, by their unwillingness to yield to the child-centred orthodoxy. It has been posted on my blog with her permission.

Dear Blogger,

I would like to thank you profusely for saving my life yesterday – or, at least, raising my spirits.

I am a rookie – NQT (FE) and have just completed three weeks’ supply teaching at a London comprehensive – my first school post – History KS3.

Yesterday, I was told not to come…

View original post 1,694 more words

A Response to Ben Goldacre’s Building Evidence Into Education Report. Part 2

Here I continue with my response to Ben Goldacre’s report on evidence in education. 

The usefulness of RCTs (Randomised Control Trials) in education cannot be determined without confronting a large number of prior questions which have largely been avoided.

1) What type of debates are to be resolved by RCTs?

It would appear that despite the excellent job he did debunking Brain Gym, Goldacre has not realised that much, or even most, educational debate is about worthless nonsense that can already be shown to be wrong long before the RCT stage. He assumes that education is like medicine was in the 1970s, whereas it is probably more like medicine in the 1370s. He assumes that we have clear aims and sound theories which need to be refined with better empirical research to identify those situations where we have been misled. However, it would be fairer to say we are at war in education over our ultimate aims and over the underlying theories. We are not 1970s doctors needing information about the effectiveness of certain drugs, we are medieval doctors trying to find the correct balance of the four humours. Teachers (and educationalists) don’t agree on aims; they don’t agree on what existing evidence shows, and they don’t always have the time and resources to implement an idea even if it is proven to be effective (for instance, look at what most teachers agree is the most effective sort of marking and what marking most teachers can actually get done in a week).

What do we actually need to be finding out? Should we be testing ideas that are obviously wrong but popular (e.g. Brain Gym)? Should we be testing ideas that are not particularly plausible to teachers but are popular with educationalists or OFSTED inspectors (e.g. group work)? Should we be testing ideas that are supported by cognitive psychology but rarely applied, (e.g. use of tests to aid recall) or ideas that are common but psychologically implausible (e.g. discovery learning)? What constitutes success for an intervention? I wrote here about the kinds of goalpost-shifting arguments we have over teaching methods and RCTs do not provide an obvious end to that debate. Whether an RCT will give useful data or waste resources will depend on resolving these issues, issues which, as a profession, we seem to be getting nowhere with. Additionally, there are ethical issues. Goldacre suggests that these are not insurmountable but “requires everyone involved in education to recognise when it’s time to honestly say “we don’t know what’s best here””. Easily said, but how many debates are there in education where everyone, or even enough people to conduct research, actually say that?

2) What are the gains to accuracy from RCTs, relative to the costs?

A lot of the arguments for RCTs assume that they are justified by being more accurate then the alternatives and that this will apply in all cases. However, this needs to be considered alongside the difficulties which make RCTs in education practically difficult. Most interventions will happen at a whole class level, making it harder to isolate the effects. We will be unable to “blind” trials (i.e. ensure that those delivering an intervention don’t know whether they are doing so or instead just delivering a placebo). We don’t have the resources of the drug companies to fund trials. I think Goldacre is right to suggest that those who say RCTs in education cannot possibly work for reasons such as these have it wrong. But these reasons do mean that before we can begin we have to accept that the level of accuracy of any given RCT might not justify the cost. We cannot simply say that as RCTs are more accurate than non-randomised trials then we can safely ignore non-randomised research even if it is abundant and overwhelmingly pointing in one direction. We are not comparing the perfectly accurate with the utterly inaccurate; we are comparing differing degrees of accuracy. In the past I have heard Ben Goldacre declare that the effectiveness of phonics is, as yet, undecided because there have not been enough results from RCTs even though there have been hundreds of non-randomised studies. This argument might work if RCTs were perfect, and non-randomised studies were always useless. It is less convincing when we admit we are actually considering imperfect data, even when we look at RCTs. How much data can we afford to throw out in the hope that RCTs will provide a definitive answer at some point in the future? We can even take this point to the extreme of asking how much more accurate RCTs are compared with the opinions teachers form from experience. It is often assumed that, just as doctors were fooled by the placebo affect or the spread of probabilities, teachers are constantly mistaken about the effectiveness of their teaching. This has not actually been established, for all teachers making judgements in all ways. Most teachers would sooner seek advice from experienced peers than reserve judgement on all matters, waiting for RCTs that may never happen.

3) What research do we have the resources for?

Underlying the previous two questions is the issue of resources. If we can conduct enough RCTs it makes it easier to choose which questions to address. If we can make the RCTs large enough, or replicate them, then we can address questions of accuracy. There is money available for RCTs, but there will always need to be rationing because debates in education seem to expand continually, and even with unlimited funding there may still be a lack of trained researchers. We need to set priorities, which is what makes it so hard to proceed without resolving the previous two questions. We also need to have some idea of the potential benefits of RCTs. Too much of this discussion does seem to present RCTs as a magic bullet, where the benefits will definitely outweigh the costs, even if we use them indiscriminately. However, we constantly have to ask which applications of RCTs will provide the greatest benefit. Again, that requires asking the previous two questions. There is no point using RCTs to resolve questions where the answer is already available, or where teachers can find a reliable answer just from experience. There is little point in investing in RCTs in areas where the results from non-randomised trials are already convincing (e.g. phonics). And what if it turns out we can get reliable results from much cheaper methods, such as training teachers to better evaluate their own practice or from testing psychological theories in the lab?

4) Who are we trying to persuade here?

There’s no getting away from the fact that teachers have learned to be sceptical of education research. Often this has been for very good reason (as I explained in my previous blogpost). Sometimes it is purely out of stubbornness. Anyone following the debate over phonics will have noticed that there is a hardcore of educationalists and teachers who simply cannot be persuaded of the benefits of phonics by reason, evidence or even direct personal experience of its effectiveness. As well as those who are irrational about methods, I mentioned under the first question that there are also a variety of beliefs about aims. It’s all very well declaring that RCTs are the best evidence, but how often are disputes in education about the quality of evidence, and how often are they about ideology? There is no point in spending money on an RCT which will, whatever it shows, be ignored by all the people who most need to be persuaded. Does anyone think that even one phonics denialist will be persuaded if, on top of the hundreds of non-randomised studies showing the effectiveness of phonics, there were a few more RCTs? Perhaps this is what Goldacre means to address by talking about “culture change” in education, however, it does leave me wondering if a focus on RCTs is like hoping a particularly accurate globe will persuade flat-earthers. The time and resources spent on conducting RCTs might be better spent on persuading teachers to accept the scientific method, or the basics of cognitive psychology, rather than looking for an unidentified degree of improvement in the quality of empirical studies.

*

Having followed some of the debate that has happened since the report came out I may be over-emphasising the issue of RCTs here. I do accept that, where practical, they may be the best form of evidence. There are certain questions, such as those about expensive interventions affecting individual students rather than whole classes, where they are both practical and suited to the task. However, there are enough practical difficulties with making good use of RCTs more widely that I do worry that pushing for them may distract from the more important debate. The real argument in education is often over the principle of using evidence rather than the question of which type of evidence. Pushing for the best possible evidence in all circumstances may turn out to make things worse for teachers faced with pseudo-science. Yes, if we set the bar for quality of evidence too low then we will be told that all sorts of nonsense is backed by research, but if we set it too high we will be told there is no evidence supporting interventions that have been shown to work time and time again. Either position will surrender territory to those who are convinced that their ideology must be enforced on the profession, and that research is to be used only to suit their pre-determined agenda rather than to ascertain the truth.

A Response to Ben Goldacre’s Building Evidence Into Education Report. Part 1

Earlier this month, Ben Goldacre’s government-backed report into the use of evidence in education was published. In this, and a subsequent post, I will respond to some of the key points and arguments.

Firstly, I will highlight the part that is to be welcomed. The suggestion is made that teaching could become an evidence based profession and that, as in medicine, evidence would aid informed decision-making. It is suggested that having expertise based on a grasp of the evidence would allow the profession to be more, rather than less, autonomous. It is suggested that teachers, by identifying the important questions from the frontline, could be the driving force in setting research priorities. There are practical suggestions, such as training teachers in research methods, finding ways to disseminate research findings and helping teachers to work with researchers.

This vision is a welcome change from the current dynamic between researchers and teachers. In my experience, the current situation is that teachers tend to encounter research in two unhelpful ways. Disastrously, dubious research is presented as the source of the latest initiatives and fads, as a reason to overrule professional judgement and embrace an idea suggested and endorsed by somebody who has either never taught full time or long since fled the classroom. In this model, research serves the interests and ideas of researchers and is used as a method of advocacy for, or an excuse for enforcement of, the latest fad. I can think of few bad ideas in teaching that weren’t, at some point, presented as the product of definitive research. Sometimes the research doesn’t really exist (e,g, Brain Gym). Sometimes the research itself is the worthless work of propagandists (e.g. Jo Boaler’s work on maths teaching). Sometimes the research is reputable but the interpretation is worthless (see countless forms of nonsense claiming to be based on Carol Dweck’s work). But the effect of this relationship has been to lower teacher autonomy and to make teachers instinctively sceptical of academics. Few teachers will change their ideas simply because of research, which is sometime an irony given that often our ideas as a profession simply reflect the work of some long since discredited researcher from a previous generation. A lot of the bad teaching methods enforced by OFSTED may have their origins in this kind of relationship between quack researchers and teachers.

The other relationship we see between teaching and research is that which broadly goes under the title of “action research”. Under this headline, some poor teacher who has foolishly decided to embark upon a masters degree in their spare time, is persuaded to carry out their own research project. Typically, this will be statistically worthless, involve lots of questionnaires and be considered worthwhile only if it shows interest in some current initiative or gimmick. While this may provide the teacher some insight into their own situation, it is unlikely to ever produce generalisable research results or anything more persuasive than the personal opinion of any other member of the teaching profession.

Overall, the “research architecture” suggested in the report is the most useful contribution to debate. The idea of a teaching profession setting the questions, and researchers investigating them, seems to be turning an upside down situation the right way round.

The less helpful discussion prompted by the report is that about Randomised Control Trials (RCTs) which are experiments conducted by applying different interventions to different people, selected at random and comparing the results. These have been particularly effective in medicine where they are used to evaluate new drugs and other interventions. In the debate over RCTs that has followed the report, I have tended to see from the anti-RCT side responses which either completely rule out evidence or RCTs, either arbitrarily, or for a reason related to a genuine difficulty but without any proper analysis of how great that difficulty is. From the pro-RCT side I have tended to see arguments which amount to little more than a confidence that problems that were overcome in medicine can be overcome here, that more trials can overcome the difficulties and the claim (without analysis) that the advantages will outweigh the costs. Goldacre acknowledges that qualitative research may be useful in explaining why certain interventions are effective (something I tend to doubt) but not that there are (quantitative) alternatives to RCTs that may prove more practical in many educational contexts.

I’m happy to accept most of the points in the report justifying RCTs as the best way to test an intervention. However, I feel that a lot of the debate in and around the paper on RCTs seems to ignore, or put off answering, some absolutely crucial questions about RCTs. These are mainly around which hypotheses are to be tested and what level of resources are to be devoted to testing them. I realise that it can be argued that these are debates for further down the road, that the first step is simply to accept the principle of RCTs, however, I think that if we fail to look at these questions first then we end up simply talking at cross-purposes for most of the discussion. What we test and how much we can spend testing it, shapes the both the usefulness and the ethics of RCTs.

In my next blogpost I will consider some of the questions that need to be considered in order to evaluate Goldacre’s case for increased use of RCTs.

This is excellent stuff from Matthew Hunter.

goodbyemisterhunter

“However specious in theory the project might be of giving education to the labouring classes of the poor, it would, in effect, be found to be prejudicial to their morals and happiness.”

Davies Giddy MP, 1807

“The academic, subject-based curriculum is a middle-class creation . . . whose effect, if not intention, has been to make it difficult for many children not from a middle-class background to adjust to a highly academic school culture.”

Professor John White, 2007

Although two centuries and the political spectrum divide these two quotations, they are united in an important sense: both deny the ability of poor children to benefit from an academic education. The first quotation comes from a Tory MP speaking against the 1807 Parochial Schools bill. The second comes from a man at the heart of today’s education establishment, an emeritus professor at the Institute of Education, University of London. His is…

View original post 53 more words

Can Sir Michael Wilshaw order OFSTED to change?

In case you’ve missed it, I’ve been banging the drum about OFSTED and their enforcement of “child-centred” education (i.e. low content/ low teacher talk lessons) and how this contradicts the claim by their chief inspector, Sir Michael Wilshaw, that there is no preferred teaching style, and how it contrasts with the comments he’s made suggesting tolerance of didactic teaching (particularly in secondary maths).

I’ve pointed out:

1. The types of lessons praised and criticised (and in what way) in recent OFSTED reports;
2. The good practice video and case studies for secondary maths;
3. The other good practice videos (now gone but not missed);
4. The way inspectors run consultancies advocating particular methods (and some further examples from reports);
5. The way inspectors seemed to be advocating that governors observe lessons (in a post written by a guest author);
6. The difference between the handbook and the subject guidance.

The first of these points, the bias towards particular types of teaching shown in OFSTED reports, appears to have been noticed by Sir Michael. According to a source within OFSTED, the chief inspector’s latest letter to additional inspectors (i.e. those who are contracted to inspection teams but not full time employees of OFSTED) contained the following section:

Of course, in practice this could mean that inspectors continue to behave in the same way, but leave less evidence of it in the reports they write, but I’m willing to take this as a positive sign that it is realised by Sir Michael that OFSTED’s habit of enforcing child-centred education is not acceptable, even if he does not acknowledge it as common practice rather than a “false impression”. It will be interesting to see if reports continue to complain of too much teacher talk and a lack of independent activities, or even better if we see reports praise traditional teaching where it is done well.

As an additional point, even if OFSTED do change, that is of little comfort to those schools who have already been criticised for not conforming to the dumbed down OFSTED model. Among the things I would still be interested in bringing to light are what people have experienced during OFSTED inspections (particularly recent ones) and what type of teaching those inspectors advocated. Please email me if you have a story to pass on.

This is such a timely exploration of the key (but largely unexplored) concept in my last blogpost, that I simply have to reblog this.

Do OFSTED pay attention to their chief inspector or their handbook?

A couple of people who I respect enormously have, during my recent OFSTED campaign, picked me up on one point: why do I write as if the organisation, and its chief inspector, have very different agendas? Or to put it another way, when Sir Michael’s words contradict his organisation’s actions why believe that it the organisation, rather than his words, which misrepresent his intentions?

My only answer really is that having heard him speak I think he deserves the benefit of the doubt on the issue of good intentions. He seems to be saying the right things as if he believed them. He was at it again this week, according to the Times (via School improvement.net )

 Sir Michael said children needed an element of rote learning, a grasp of basic facts and to master reading, writing, spelling, punctuation and grammar before they could learn at a higher level, and many prep and private schools used a similar approach.

As a head teacher he had long seen a need to replace the national curriculum with one that emphasised such a more traditionalist approach, especially in maths and English, he said.

“I am extremely upset and concerned that there should be this level of criticism for what I think is absolutely essential – more rigour in the national curriculum and a greater focus on basic skills,” Sir Michael said.

However, no matter how good his intentions might be, it is worth asking what difference anything he does makes. Here I want to look at one example of OFSTED at the centre (presumably under Sir Michael’s direction) making an important central change which is then immediately ignored by other parts of the organisation. As I’ve discussed before, the old OFSTED handbook contained the following phrase in its description of “outstanding” teaching:

 Teaching promotes pupils’ high levels of resilience, confidence and independence when they tackle challenging activities.

For good teaching:

 Teaching generally promotes pupils’ resilience, confidence and independence when tackling challenging activities.

This could be interpreted in many ways but in education “independence” has become code for minimising the role of the teacher and the amount of direct instruction. This is something that, perhaps, may only becomes apparent when you hear teachers say apparently contradictory things such as “I do lots of groupwork in order to encourage independence” and I realise some may doubt that this is the usual interpretation (although it can be argued that guidance at the time made clear it was the correct interpretation). What cannot be doubted is that when the post-Wilshaw version of the OFSTED handbook appeared in late 2012 this phrase was removed and the only refernce to “independence” was:

Not all aspects of learning, for example pupils’ engagement, interest, concentration, determination, resilience and independence, will be seen in a single observation.

Now, whatever your precise understanding of the text, this seems to be a significant change from requiring the promotion of independence as a feature of outstanding teaching, to only mentioning it as something that does not need to be seen in every observation and it appeared to underline the message Sir Michael had communicated that there would be no OFSTED-approved teaching style. I have certainly seen the old text used in schools as “proof” that OFSTED require a particular method of teaching, and disappointment on the part of true believers when they discovered that text was no longer a canonical part of the Gospel according to OFSTED. We can assume that this could not have happened by accident or against Sir Michael’s wishes and that anyone claiming that OFSTED are still looking for “independence” was behind the times and anyone specifying that this must include groupwork must be unaware of Sir Michael’s repeated claims that no particular teaching style was required.

For these reasons, a good test of how much this kind of central change makes would be to see whether it has affected the subject guidance documents. These materials are to be used when surveying the quality of teaching of individual subjects in schools, providing more detail and at times providing a more detailed interpretation of the general criteria and forming the basis of any subject reports. These were updated in the early months of this year, and because they are written by subject specialists and so provide a good indicator of how much effect the OFSTED handbook changes have had across OFSTED. If we look to see what these say, after the requirement for independence was removed from the handbook, we find how little things have really changed under Wilshaw.

From the science guidance we find that outstanding achievement requires:

Pupils show exceptional independence; they are able to think for themselves and raise their own questions about science knowledge and understanding and scientific enquiry.

Good achievement means:

Pupils regularly work independently, often taking the initiative in individual work and when working with others.They show confidence and competence in the full range of stage-appropriate practical work, including planning and carrying out science investigations in groups or individually

Achievement which requires improvement means:

 Pupils are generally dependent on their teachers, particularly when the teaching methods used do not encourage independent thought.

Inadequate achievement means:

Pupils rarely work independently or take the initiative in their work.

From the English guidance, we find out that outstanding achievement means:

Pupils have learnt to be effective independent learners, able to think for themselves and to provide leadership, while also being sensitive to the needs of others.

Good achievement means:

Pupils express their ideas clearly and well in discussion and work effectively in different groups. Pupils are able to show independence and initiative; for instance, raising thoughtful questions or helping to drive forward group work.

An outstanding curriculum means that:

Independent learning and wide reading are very well promoted.

A curriculum in need of improvement means there are only:

Some opportunities are provided for pupils to work independently.

In history, inadequate achievement means:

Pupils rarely demonstrate enthusiasm, initiative, creative or the ability to learn independently in history.

In RE, outstanding achievement means:

Pupils show exceptional independence; they can think for themselves and take the initiative in, for  example, asking questions, carrying out their own investigations, evaluating ideas and working constructively with others.

Good achievement means:

Pupils show independence; they can think for themselves and take some initiative in, for example, asking questions, carrying out investigations and working with others. 

And inadequate achievement means:

Pupils rarely show the ability to work independently or take the initiative in RE.

In modern languages, outstanding achievement means:

Pupils show exceptional independence in their studies and can use a range of resources, including ICT, to develop their language skills and investigate aspects that interest them.

Almost all pupils work hard, develop resilience and understand that language learning is often challenging, purposeful and collaborative.

Good achievement means:

Pupils are able to work independently when given the opportunity, taking the initiative in their work and when working with others.

Achievement requiring improvement means:

Pupils can occasionally work independently and take initiative in developing their work but more often are dependent on their teachers for written and oral prompts when trying to create new sentences….

…Some pupils are reluctant to work in pairs or groups using the target language and frequently return to English.

Inadequate achievement means:

Pupils are unable to work independently or take the initiative in their work.

Outstanding teaching means:

Precisely targeted support from other adults encourages all pupils to develop independence and a desire to use the target language for real communication.

Good teaching means:

Planning is informed by a good level of subject expertise. As a result, teachers use an appropriate range of resources and teaching strategies to promote good learning across all aspects of the subject and ensure pupils develop the skills they need to become independent language learners.

An inadequate curriculum means:

Pupils are given insufficient opportunities to develop creativity, linguistic competence, cultural understanding or the skills needed to be independent language learners.

Even in PE, outstanding achievement means:

They know how to improve their own and others’ performance, and work independently for extended periods of time without the need of guidance or support.

Achievement requiring improvement means:

Pupils are too dependent on the teacher and cannot work independently for sustained periods of time without their support or guidance.

Inadequate teaching means:

Too much teacher talk, low expectations and few opportunities to learn independently lead to long periods of inactivity.

In maths, outstanding achievement means:

Pupils … show exceptional independence and take the initiative in solving problems in a wide range of contexts, including the new or unusual. Pupils think for themselves and are prepared to persevere when faced with challenges, showing a confidence that they will succeed.

Good achievement means:

They are able to work independently, and sometimes take the initiative in solving problems in various contexts.

Outstanding teaching (in a section which also contradicts Wilshaw’s previously indicated views about maths teaching) means:

Teaching is rooted in the development of all pupils’ conceptual understanding of important concepts [so much for Wilshaw’s “element of rote”] and progression within the lesson and over time.

…Teachers nurture mathematical independence, allowing time for thinking and encouraging discussion. Problem- solving, discussion and investigation are integral to pupils’ learning of mathematics…

 … [Teachers] use a very wide range of teaching strategies to stimulate all pupils’ active participation in their learning, together with innovative and imaginative resources, including practical activities and, where appropriate, the outdoor environment.

Good teaching means:

Teaching develops pupils’ understanding of important concepts as well as their proficiency in techniques and recall of knowledge, equipping pupils to work independently.

In EBE (Economics, business and enterprise), outstanding achievement means:

Pupils on formally assessed economics and business education courses show exceptional independence; they are able to think for themselves and take the initiative in, for example, asking questions, carrying out their own investigations and in working constructively with others. They show significant levels of originality, imagination or creativity in their understanding and skills within the subject.

Good achievement means: 

Pupils on formally assessed economics and business education courses are able to work independently when given the opportunity, taking the initiative in their work and when working with others. They demonstrate some originality, imagination or creativity in their subject work.

Achievement in need of improvement means: 

Pupils on formally assessed economics and business education courses are generally dependent on their teachers but can occasionally work independently and take the initiative in developing their work.

 Inadequate achievement means:

Pupils on formally assessed economics and business education courses rarely show the ability to work independently or take the initiative in their work. They rarely demonstrate creativity or originality in their subject work.

In geography, outstanding achievment means:

Pupils show exceptional independence; they are able to think for themselves and take the initiative in, for example, asking questions, carrying out their own investigations and working constructively with others. They show significant levels of originality, imagination or creativity in their understanding and skills within the subject.

Good achievement means:

Pupils are able to work independently when given the opportunity, taking the initiative in their work and when working with others. They demonstrate some originality, imagination or creativity in their subject work.

Achievement requiring improvement means:

Pupils are generally dependent on their teachers but can occasionally work independently and take the initiative in developing their work. Occasionally, pupils show creative or original responses in their subject work.

Inadequate achievement means:

Pupils rarely learn independently and rely heavily on the teacher to provide answers.

In music, inadequate achievement means:

Pupils rarely show the ability or willingness to work independently or take the initiative in their work.

Now I realise there is a lot of ambiguity about the word “independence” and it is easy to find interpretations of the word that everyone would be quite happy with. But what is the point of removing a word from the OFSTED handbook other than to say it will not be expected to be seen in every observation, only to set out guidance a few months later suggesting it is a core attribute of what achievement looks like in almost every single subject? Who is really setting the agenda here?

How does OFSTED help failing schools?

Apologies for excessive reblogging, but if you read my blog, you need to read this one too.

This is good. I hope I’m not overdoing the reblogging.

The next one of these. They’ve all been interesting.