A Note on Exams

Formal assessment is not a subject I have ever studied in any detail and I have only had reason to think about it in detail because of the bizarre arguments over English GCSE. These have made me realise how differently some people see exams. So, to begin with, I think it’s worth going over these fundamental issues, but be warned, this is entirely based on personal experience of the debates and no actual research on the theory or practice of formal assessment.

As far as I am concerned, exams are to test ability. I accept that it is both inevitable and perfectly acceptable that they will reflect effort as well and when I talk about ability in this blog, you can assume that I have accepted the point that this actually means ability mediated by effort. This, however, is all that they should be looking for. If an exam allows comparison of ability then it functions. If outcomes become sufficiently detached from ability then the system is breaking down.

Now an exam system requires standardisation, a way of comparing ability levels even when people have taken different exams. If no way is found to do this from one year to the next then we have grade inflation (in theory we could also have grade deflation, but that does not actually seem to happen). The key priority, therefore, has to be that the difficulty involved in getting a particular grade is consistent between exams. This, perhaps obvious, point is often missed in debate over exams. People talk as though they can identify what deserves a C without identifying how difficult it might be to pull off at a particular time or in particular circumstances or what level of genuine ability it will require. Debate becomes easier once we accept that it is difficulty that must remain constant. And this is not just about results from year to year; within our system we also require standardisation between exam boards and tiers of entry. There are several different approaches to these issues.

The classic method is norm-referencing. This is where roughly the same proportions get each grade each year. This does not really help us standardise between tiers or boards, but it is objective and does help address standards over time. The problem is that is does not work if genuine changes in cohort ability occur. People often use this as an excuse to dismiss it out of hand. Countless people have asked questions along the line of: “well what if everyone simultaneously became as smart as last year’s A* students?” Yes, this would make the effects of norm-referencing unfair, but it is a bizarre way to approach anything. You might as well ask an economist if their measures of inflation would work if everyone spent all their money on radishes or complain that a thermometer wouldn’t work if you threw it into the heart of the sun. It is no argument against anything that it wouldn’t work in an impossible situation. The fact is that ability levels don’t change much from year to year. There are, nevertheless, difficulties that may occur. Firstly, in a complicated exam system there may be changes in the cohort entered for particular exams. This is best dealt with by looking at other data, say KS2 results, so as to adjust results. This is the basis of the “comparable outcomes” approach used in our exams this year. Secondly it does not give a huge amount of guidance for comparing between different exam boards and tiers. Again, other data like KS2 scores can be used. Thirdly, it is likely to be less effective over the long term. While there may be little change in cohort ability from one year to the next, there are likely to be changes from one decade to the next. Even IQ scores change on that basis. Despite this, if other data is used to check for changes in ability, it is the most objective method of maintaining the level of difficulty, and the easiest method to use in the short-term.

There are a couple of alternative approaches which seem to make sense but actually do not maintain consistency in the level of difficulty and are behind a lot of the problems with exams. Firstly, we have the idea that there is a particular performance that simply has to be replicated in order to get a grade. This seems to make intuitive sense and its advocates will usually use a series of analogies. The 100m world record will always be for running 100m. There will not be days when it is okay to run a couple of people over on the driving test. These are simply tests of performance, not difficulty. However, there are key differences. Both athletic world records and driving tests set a simple standard which you only need to beat once. If circumstances make them difficult to achieve on one occasion, then you can do them again. The standard is one to be met on your best day and so it doesn’t matter if, say, the weather makes it harder to achieve on one particular day. Exams are not like that. Expense and the numbers involved mean that there is a huge problem with having an exam where everybody is disadvantaged, whereas there is no problem in a 100 metres race where all the competitors are performing well below their personal best because of conditions. Even more importantly, it is in the nature of intellectual accomplishments that they involve more than repetition. Watching Usain Bolt run will not make you a world record breaker. Watching somebody else pass a driving test will be of only marginal advantage to helping you pass yours. Watching somebody else answer an exam question tells you how to answer it (providing you understand what they wrote). This is why questions change each year. There can never be a simple, repeatable grade C performance, if there was then it would become easier to achieve the more it was repeated and the more information about it was released and this would erode standards over time. Despite this, I have repeatedly heard people defend grade inflation on the grounds that, as teachers have been able to better prepare students for exams (i.e. teaching to the test) then grades should go up to reflect this better performance. This is to miss the importance of measuring ability.

The second problematic alternative to norm-referencing is to have a list of criteria which have to be met for a grade. This is very common and is embedded in the National Curriculum which has some pretty arbitrary attempts to match grades with criteria. This box-ticking approach is unworkable for two reasons. Firstly, the same criteria can be met in difficult or easy ways meaning they do not preserve difficulty. There are difficult and easy books to read and understand. There are difficult and easy long multiplication questions. There are difficult and easy analyses of historical events. Criteria that were capable of distinguishing between all levels of difficulty within a topic would be unmanageable. Secondly, if the initial criteria are botched then they tend to be ignored. So for instance, the maths National Curriculum says “mental recall of multiplication facts up to 10×10 and quick derivation of corresponding division facts” is Level  4, i.e. the level of the average 11 year old, which I believe is equivalent to grade F. However, if you ask a few 16 year olds for the factors of 56, you will quickly realise that it is not actually achieved by even the average 16 year old, and that includes many with grade C in maths. Criteria are convenient lies used mainly by those who don’t believe in grades and levels and would sooner have endless lists of what students can or cannot do used for assessment. Nevertheless, the myths that exams measure the genuine meeting of criteria, or the repetition of a particular performance, have been trotted out again and again by the apologists for grade inflation making sensible debate about what is going on in the exam system impossible

Finally, there is another method that works. This is getting experienced and academically able teachers to look over the exam and judge how difficult it is. I suspect this, rather than norm-referencing, is actually the key to those exams in the world that have maintained their standards even those that claim to be criteria-based. It is the only thing that is going to make a difference over time. The subjective judgement of the genuine expert is probably the gold standard when it comes to maintaining the level of difficulty in exams. The problem is that, while some exams taken by small numbers of students might be managed by experienced experts, the mass of the exam system, particularly the GCSEs like maths and English which everybody takes, are not managed in this way. This is because we have an education system where authority does not depend either experience or academic ability. There are people running schools, quangos and consultancies or working in education departments at universities, who will swear on their mothers’ lives that an easy question is hard and a hard question is easy. In this respect the inability to maintain standards in exams is a symptom of the inability to maintain professional standards in a bureaucracy. This may be the root cause of our current difficulties and until it is resolved then a variation on norm-referencing, such as “comparable outcomes” is the best we can hope for.

Advertisement

A New Summing Up

Every so often I like to try and sum up my overall thoughts on the education system, so as to help sum up the message, and future direction, of this blog.

The central contention of this blog is that our state school system is simply not good enough. It does not provide a decent quality of education for the vast majority, and most would avoid it if they could afford to. Too many people with power over education are content to provide a service that they would not think good enough for their own children.

There are three key issues. The first is the dumbing-down of the curriculum and teaching methods. The second is the Behaviour Crisis. The third is bureaucratic and incompetent management. I will deal with each in turn.

1) Dumbing-Down.

The most widely publicised form of dumbing-down is grade inflation. Exams have become significantly easier over time. It has resulted from so many different types of change (content, grade thresholds, predictability of questions, modularisation, early entry, choices in what content can be followed) over so many years that the only way to really see it is to actually sit down and look at old papers and mark schemes. This has meant that while virtually every experienced teacher knows exams have got significantly easier, and multiple studies for individual subjects have catalogued it, those who wish to deny it need only declare that when they look at the exam papers they, personally, don’t see it. However, it is now acknowledged by the exams regulator Ofqual and by both government and opposition front benches. The second most widely publicised form of dumbing-down was the shift in the last ten years towards “equivalent” qualifications. Worthless “vocational” (i.e. non-academic) qualifications, mainly assessed by coursework (i.e. done by cutting and pasting from the internet or with help from teachers) were given inflated values, sometimes equivalent to 4 GCSEs. This encouraged schools to opt out of proper subjects and then claim credit for improving results. Progress has been made on these two issues, with the introduction of the Ebacc; the revaluation of vocational qualifications, and an apparent effort on the part of Ofqual to confront the problem.

However, to a large degree these problems are only the symptom not the cause. The bigger problem is an anti-academic ethos in schools that allowed these things to happen. Since the turn of the twentieth century there has been a “progressive” movement in education which has sought to minimise the direct teaching of knowledge. The grounds for doing so vary dramatically. Sometimes progressives seeks to suggest that the aim of education is something other than intellectual improvement, and puts forward socialisation, therapy, enjoyment/happiness, or social and political change as alternative aims. At other times they seek to deny that intellectual improvement is mainly about knowledge, and suggests the teaching of skills, attitudes or vague attributes such as “creativity”. Or they may make claims about pedagogy, such as claiming that children learn best from being manipulated or entertained, or by being left to work things out for themselves, which seek to minimise the need for teaching. Regardless of the precise argument used (and there are many, many arguments) the end result is a classroom where instruction is replaced by activities (usually in groups) and effort on the part of students is minimised. This has been dominant in UK states schools since the sixties and, despite various attempts by public and politicians to undo the damage, (often, unfortunately, based on centralisation or the idea that examination system can be used as a tool for change) it still remains the orthodoxy in schools to the point where it is considered controversial, contrarian or provocative to actually challenge the basic doctrines of progressive education. Every mechanism that government has attempted to use to challenge the hold of progressive education – for instance: the National Curriculum, National Curriculum tests, OFSTED, the National Strategies – has invariably been captured either immediately or over time by the progressives and used to strengthen their hold on the system. Every attempt to reduce the power of particular groups identified as being behind progressive education (e.g. unions, Local Authorities, teacher training institutions) has had no effect at all as they continue to exercise influence from elsewhere (quangos, private consultancies, SMT).

2) The Behaviour Crisis.

If children are going to learn then it is absolutely vital that they do what they are told in lessons. If schools are going to be safe and orderly then it is essential that they also do what they are told outside of lessons. If teachers are going to be effective then they cannot be constantly faced with the stress of confrontation, defiance and chaos. The minimum standard required for effective teaching is that all teachers (not just SMT or teachers who have been around for years) can expect students to comply with all of their instructions first time. The minimum standard for teaching to be a desirable profession is that teachers have freedom from fear when it comes to giving instructions and enforcing rules. Too many schools simply do not have those standards, and as a result teaching is very often stressful and unpleasant.

At the heart of the problem is, again, the progressive education ideology. If education is not about instruction in knowledge then teachers are not there to be listened to or to be authorities. If the education experience is to be characterised by entertainment, inspiration and pleasure, with no need for hard work, then students will never want to thwart it. If students learn best when left to their own devices, then they will not need to be told what to do and the more freedom from external constraint they have the more they will learn. The progressive rejection of a tradition of knowledge to be passed on can be turned into an implicit moral judgement:  traditional teaching is inherently immoral and children, if uncorrupted by it, are inherently good.

This implicit belief in the saintliness of children lies at the heart of many contemporary developments in education. Any apparent wrongdoing by children must have a cause other than the moral failings of the child. Their teachers must have provoked it and should be blamed for it and never supported. They must have a condition of some kind that made them do it (this assumption underlies much of the SEN system).  They must be reacting to a harsh and unkind life, and, therefore, need to be treated with affection and compassion which will change them, rather than punishment that will make life worse. Even the worst behaved children should not be excluded. All punishment is inherently  suspect.  Additionally, as children are basically good, their wants must actually reflect underlying needs, not selfish desires. They should get what they want. They should be consulted on matters related to the running of the school. Adult authority is not to be supported.

As with dumbing-down there are grounds for optimism. There has been a significant shift within the education system. It is now much more common to hear people recommend zero-tolerance discipline policies (i.e. actually punishing kids when they do something wrong) and there are celebrated success stories where such an approach has been taken, of which Mossbourne Academy is the most famous. A few years ago it was a given that schools were meant to comply with a principle of “inclusion” which meant keeping the worst kids in mainstream schools and mainstream classrooms within their schools no matter what they did. There are also signs that teachers are more willing to take industrial action where discipline isn’t enforced properly. However, progress is slow and many schools are as bad as ever. While “inclusion” is no longer the buzz-word it was, the new buzz-word is “engagement” and the argument is frequently made that behaviour is best managed by “engaging” students, an argument which ignores the need to have high expectations at all times, and replaces it with pressure on teachers to entertain or appease.

3) Management Failure.

Schools are bureaucracies. By that I mean they are systems which engage in pointless and unnecessary activity rather than concentrating on their core purpose. Again, the ideology of progressive education plays a role.  Once schools lose sight of the fact that they are there to educate then other types of activity will multiply and additional infrastructure will be created to manage them. Some of the progressive education ideas, like replacing exams with coursework or teacher assessment do create more paperwork, as does widening the scope of assessment to include more than academic achievement.

However, this time there are at least another two significant causes. As concern about standards has (understandably) increased, politicians have been all too keen assume that what is missing is accountability. A culture has been created where paperwork is created entirely to please those who will scrutinise you. For teachers, this means their managers now have power to demand all sorts of extra written work (lesson plans, marking, assessment records) beyond what is necessary to look for actual neglect or incompetence. For schools, it means managers are preoccupied with producing policies, assessments, and “evidence” for OFSTED. What was meant to ensure that people were doing their job has actually ensured that they aren’t because they are too busy creating a paper trail. Further pressures from outside have turned exam entries from a routine piece of paperwork to a prolonged exercise in strategy, deciding exactly which type of exam from which exam board will do most to boost results.

The other cause is the structure of management. Management teams have grown to ridiculous sizes. SMT have been encouraged to be “leaders” rather than managers, which seems to mean identifying trendy initiatives (often based on forcing teachers to adopt progressive education methods) while actual management of both staff and students has been increasingly delegated downwards. Groups of teachers who would once have been teams, are now divided into managers and managed. If you are a classroom teacher, a head of department is no longer a more experienced colleague who will give you support when you ask; they are a boss checking up on you, telling you how to teach and chasing you for paperwork. The more minor positions in the school, which once were about getting paperwork (like exam entries or schemes of work) out of the way are now about creating more paperwork and pressure for staff to do. Workload, always an issue, has now become an insurmountable obstacle leaving many teachers concerned only with faking compliance with the pressures from above so as to cope at work, rather than putting their time and effort into their classes.

Keeping up appearances and finding someone to blame when you can’t, is now a far higher priority than effective teaching or successful learning. With so many rungs on the management ladder, and so much pointless activity, there is no accountability for managers who will change responsibilities long before their performance can be judged. We now have an enormous “management class” who are overwhelmingly unable to manage and often barely able to teach. Often it is truly shocking how bad at teaching managers are, not to mention how intellectually limited and prone to bullying some of them are.  Even though managers are not promoted for being good teachers, they are still, nevertheless, given responsibility for telling other people how to teach. Ideas about management have also got more and more confused, with an increasing focus on leadership (i.e. vision and setting direction), something which, while desirable in a leader, is useless when nobody is actually managing the effective day-to-day running of the school.

Unlike dumbing-down and the behaviour crisis, there is no reason for optimism about management failure. The government seems convinced that it is classroom teachers, not managers, who are to blame for falling standards and low aspirations and seems determined to increase the power of managers while worsening conditions for those on the frontline.

Of course, you can always just pretend that nothing I describe is actually happening