The Proxy Trap: how poor are 'poor proxies for learning'?

Why schools can't live with proxies or without them

Less than a decade and a half ago, teachers in England could be branded as being inadequate on the basis of a single lesson observation.

The lesson might be carefully explained, intelligently sequenced and understood well enough by students, but if the wrong person walked in at the wrong moment or if students didn’t visibly grasp a new concept or master a new skill in less than twenty minutes it was assumed the teacher was at fault. This was the era of bizarrely convoluted and unrealistic demands for differentiation, multiple mini-plenaries where students were expected to constantly demonstrate their learning instead of actually, you know, learning anything, compulsory group work and the inexplicable obsession with ‘pace’. If lessons were too quick or slow slow they were unsatisfactory but what pace was just right? No one knew. Essentially, it was just vibes.

Lesson observation had become one of the profession’s strangest rituals: half quality assurance, half theatre criticism, half séance. Observers entered classrooms for a brief period and then spoke as if the hidden life of learning had disclosed itself. Teachers prepared not just to teach, but to be seen teaching. The lesson had to communicate its own effectiveness to someone who often didn’t know the class, didn’t know the curriculum as the teacher knew it, and would be gone before the most important consequences of the lesson could possibly appear. But no one had really questioned the need for any of this. It was just the way it was.

I’d been worrying away at this on The Learning Spy for several years before “poor proxies” became part of the common vocabulary. In 2011 I’d asked what lesson observations were for; in 2012 I’d asked whether teacher observations were a waste of time; by 2013, the concern had hardened into something more serious. Lesson observation, at least as commonly enacted, distorted teaching, pushed teachers to focus on performance rather than learning, and created a system more interested in short-term fluff than real improvement.

The pre-2013 classroom was shaped by a whole ecology of inspectable improvement. Ofsted’s shadow was long, but it wasn’t only Ofsted. National Strategies, perverse Assessment for Learning rituals, APP grids, graded lessons, , three-part lessons with starts, mains and, er, puddings, VAK, sub-levels, and a whole raft of other unevidenced mumbo jumbo all combined to instil a very relief belief that learning could and should be made visible. Famously, John Hattie wrote a very influential book about the concept of Visible Learning.

Visible Learning, hidden error

Teachers learned to adapt, to play the game. If observation forms asked for visible progress, teachers provided visible progress. If book scrutiny rewarded written feedback, written feedback multiplied. If school policy prized student activity, activities were arranged. What may have begun as a set of suggestions ended as instructions.

The mythology was remarkably durable. Although Ofsted insisted it had “no preferred teaching style,” schools, rationally, didn’t believe them. In a post from December 2013 (in which I correctly predicted the imminent demise of grade observations) I noted the absurdity that schools with vulnerable data were especially exposed to the tastes of individual inspectors, and that too many schools were doing things that weren’t required by the framework because they didn’t know what kind of inspector they’d be saddled with.

Lesson observation had become more than a way of seeing teaching; it shaped teaching. If teachers thought an observer wanted less teacher talk, they spoke less. If they thought progress was expected to be made visible, they inserted a meaningless progress check. And if they thought independent learning was expected, they spent hours putting together nonsensical group tasks which — all too often — interfered with the teaching ont heir subjects. When accountability rewards compliance, teachers will — if they can — comply.

Although manyu of us knew something was wrong, we lacked the language to say so without sounding defensive. We might have instinctively felt a lesson could dazzle and leave little behind. Or a lesson which looked slow, awkward might be be doing more intellectual work than one fizzing with group work and lolly sticks.

In Rob Coe’s 2013 lecture, Improving Education: A Triumph of Hope Over Experience, he listed familiar classroom signs that were easily observed but not necessarily about learning: students being busy, especially producing written work; students being engaged and motivated; students receiving attention, explanations and feedback; classrooms being calm and controlled; curriculum being covered; and some students producing correct answers, regardless of whether they understood, could reproduce those answers independently, would remember them later, or had already known them.

It was a devastatingly simple list. Coe wasn’t saying that engagement, order, feedback, coverage, written work or correct answers were worthless — in fact, he went to pains to make clear this wasn’t the case. His argument was that such activities couldn’t bear the weight of conclusions being loaded onto them. He wasn’t attacking visible evidence, but the over-claiming that came bindled with them.

Schools had become too confident that learning could be recognised by its surface features. The notion that such things were ‘poor proxies for learning’ gave teachers and leaders a way to ask a question that had beenneglected for far too long: are we sure this means what we think it means?

The later evidence on lesson grading made the point harder to ignore. In 2014, I wrote about the reliability problem in graded observations: if one observer judged a lesson outstanding, the probability that a second observer would give a different grade was between 51% and 78%. If a lesson was judged inadequate, there was a 90% chance that a second observer would grade it differently. If observation judgement depends so heavily on who happens to walk through the door, we should be wary of building careers, reputations and school improvement plans on it. The issue wasn’t that observers were stupid or malicious but that they were being asked to infer too much from too little.

It’s hard now to recreate the force of that moment because so many of the old orthodoxies have happily been swept away. Few people now defend the graded twenty-minute observation or the idea that progress must be visible every lesson. Ofsted later drew a line under grading individual lessons, and the profession became more comfortable saying that teachers were allowed to talk, explain and teach.

But these myths are like bindweed. It’s not enough to root them out once, we have to contintually guard against them growing back. Visible proxies are attractive because learning is devilishly hard to perceive. Learning is slow, internal, fragile and often unavailable at the point we want evidence of it. learning, to be worthy of the name, must be flexible and durable. It must last beyond the narrow slice of the present. But, obviously, all we get to see is the here and now; we have to make guesses about elsewhere and later. An observer visiting a classroom can see behaviour, hear answers, glance at books, note participation and feel the mood of the room. What they can’t see is whether what has been taught will still be available next month, in a different context, without the teacher’s cues.

Compared with that stubborn invisibility, proxies are wonderfully compliant. They can be written into policies, and checked. That, I think, is the enduring value of Coe’s argument: it didn’t tell us to stop looking but stop being so easily satisfied by what is most easily available.

But recently, Jon Hutchinson made an interesting challenge in a comment on LinkedIn about a post I’d made about the danger of over-relying on presentation stanrds in students’ exercise books as a proxy for learning.

As a counterpoint, it’s depressing for many primary teachers to insist on children completing their work to a high standard of quality, secure in the knowledge that when they get to secondary this will often be allowed to atrophy really quite rapidly. As you say, there’s nothing clever or humane about allowing children to scrawl illegibly, but over time this is often what we see. Sure neat presentation isn’t a proxy for learning, but neither is sloppy presentation a proxy for thinking.

fwiw, I am increasingly of the opinion that Coe’s poor proxies for learning is one of the most damaging ideas in education. It moved rapidly from ‘these things don’t guarantee learning’ (true) to ‘so it’s not important to focus on them (speculation) to ‘they should be avoided’ (nonsense)

Good ideas are so easily reduced to unhelpful slogans. If Jon’s right, “engagement isn’t learning” may have morphed into, “engagement doesn’t matter”. “Although none of this follows from Coe’s argument, schools rarely run on logical entailment. Slogans, incentives and half-remembered CPD are more commonly the mood music of the moment.

Maybe phrase “poor proxy” is too self congratulatory? Maybe pointing out the differences between performance and learning is too smug to be practical in the hurly burly of the classroom? Maybe the phrase has becomea way of ending conversations rather than spurring deeper thought?

There’s some cultural evidence for this, even if the case is difficult to prove empirically. In blogs, on social media, in CPD providers’ materials, “poor proxies” has sometimes been used as a badge of being ‘in the know'. Engagement has come to be treated with suspicion and enjoyment as embarrassing. In the name of ‘raising standards’ and ‘knoweldge rich,’ a more austere age of education was — unintentionally — ushered in.

Did the poor proxy kill off one theatre of teaching, only to create another? Post Coe, at least in some schools, the approved performance has become retrieval starters, cold calling, silent compliance, knowledge organisers, cognitive load slogans, mini-whiteboards and everyone “thinking hard”. These are, at least in theory, better than what went before but, in practice, are just as prone to becoming empty rituals. The point isn’t to identify which is worse but that we should acknowledge that ritualised teaching is never useful if no one knows why they’re doing what they’re doing.

We have a remarkable talent for turning correctives into costumes. We’ve torn down the false idols of pace and engagement and instead begun to worship effort and rigour. The idols may have changed but the idolatry remains.

Obviously, a teacher who cares whether students are interested isn’t shallow. A teacher who wants students to enjoy the lesson isn’t necessarily pandering. Values relationships doesn’t mean the curriculum has been forgotten. And, I’m forced to acknowledge, a school caring about presentation, confidence, participation and atmosphere may also be attending to real conditions that make learning more likely.

There is a deadening version of evidence-informed education which seems embarrassed by anything too human. It can make warmth sound soft, enthusiasm sound unserious and enjoyment sound like a guilty secret. It can forget that students are not disembodied working memories waiting to be loaded but social, anxious, status-conscious, effort-avoiding, approval-seeking, meaning-making creatures. They learn through attention and memory, certainly, but attention and memory are affected by trust, belonging, confidence, interest, clarity, success and the emotional weather of the room. If “poor proxies” are used to dismiss all that, then they’ve become poor thinking.

The trouble is, all this is prone to infinite regress. The criticism of proxies can itself become a proxy. The person least impressed by engagement and most suspicious of enjoyment can claim to be the most rigorous. Teachers who strive for visible struggle appear more intellectually honest than those whose students are succeeding fluently. But fluency can be shallow and struggle can be stupid. Students can think hard about the wrong things. They can think hard about deciphering instructions, managing anxiety, second-guessing the teacher, navigating classroom dynamics or enduring confusion. Hard thinking is only useful when it’s about the right content and is likely to leave useful residue behind.

The distinction between learning and performance helps, but doesn’t solve everything. Immediate performance can mislead us. Students can succeed now because the model is still visible, the teacher’s prompts are fresh, the task is familiar, or the answer has been smuggled into the question. Equally, making things harder doesn’t magically make learning better. Desirable difficulties have to be desirable. Undesirable difficulties are just difficulties. Forgetting this gives us a new proxy: if it looks effortful, it must be learning.

A blood pressure reading, cholesterol level or test result can be vitally important, but the numbers aren’t a patient’s health. A doctor who ignores the numbers is reckless but one who treats only the numbers is also reckless. Measures matter because they points beyond themselves. They become dangerous when they replace the thing they were meant to illuminate. If schools ignore proxies they make themselves wilfully blind, but worshipings proxies is a direct route to a different kind of blindness.

For instance, used well, book scrutiny can tell us a great deal. It can suggest whether students have had enough practice, whether the curriculum is coherent, whether writing is improving, whether misconceptions recur, whether tasks are worth doing, and whether students are developing fluency over time. But, as Campbell’s Law tells us, once everyone knows what books are supposed to look like, they become increasingly meaningless. The artefact improves and the evidence worsens.

Lesson observation follows the same logic. Most observers are looking for sensible things: clarity, participation, checking for understanding, routines, behaviour and challenge. But the visible texture of the lesson can flatter us. A lesson can appear fluent if it’s over-scaffolded. It can seem demanding if students are busy. It can look successful if the questions are too narrow and calm because very little is being asked.

If students can do the thing now, surely they have learned it? Sometimes, yes. Often, not yet. All instruction begins with supported performance, so there’s nothing wrong with students succeeding with cues, models and prompts. The mistake is treating that supported performance as the final claim rather than the beginning of one.

This is why I still think the argument against poor proxies is stronger than the argument against the argument. Jon is right that the phrase can be misused. He’s right that some people have used it to sneer at things that matter and that the evidence-informed world has replaced one set of inspectable superficialities with another. But the belief that proxies are inherently bad hasn’t built entire accountability systems. The thoughtless worship of poor proxies has.

Proxy worship writes policies, designs learning walk protocols, produces book scrutiny pro formas and shapes appraisal targets. It fills CPD calendars, tells teachers what will be approved and students what success is supposed to look like. It creates the conditions in which everyone can work hard to improve the appearance of learning while learning — the retention and transfer of what we what students to master — is ignored.

That was true before 2013, when schools were seduced by the visible signs of progressive busyness and it can still be true now, when schools are seduced by the proxies of evidence-informed seriousness. The costumes have changes but the appetitie for easy reassurance remains.

The lesson, then, is that the poor proxies argument has to be turned on itself. “Thinking hard” can be a poor proxy, as can retrieval practice, silence, cold calling, knowledge organisers and all the rest. A school can wear the trappings of rigour and still leave students understanding little.

The better position is to embody “proxy humility.” Look at books, but don’t be seduced by what you see. Notice engagement, but ask what students are engaged in. Value calm classrooms, but ask whether calm is serving thought. Care about accuracy, but distinguish immediate performance from durable learning. train staff to use assessment routines, but remember that every assessment samples only part of the domain. And continue to watch lessons, but resist the temptation to believe that what you notice is the same as what students learn.

We should always distrust easy inferences but even scepticism can become lazy. We need proxies but we should never rely on them. The enemy was never engagement, order, feedback, written work or any other visible trace of learning. The enemy has always been our insatiable hunger for easy answers.