The best case fallacy and the need to audit optimism

Why optimism is often unscrupulous and positive thinking makes things worse

Optimism, n. The doctrine or belief that everything is beautiful, including what is ugly.

Ambrose Bierce, The Devil’s Dictionary

Open any social media app and you’ll be inundated with injunctions to manifest success, trust the process, believe in yourself and never allow negative thoughts stand in your way. Good things will be yours if only you believe hard enough.

There are even people who tell us the reverse: that all the bad things that happen to us are the fault of our own negativity. This is, I’m happy to say, bullshit. Dangerous bullshit: I know I haven’t revised for my exams, but if I believe in myself strongly enough, I’ll definitely get an A. Or, I’ve just had a few pints at the pub and I feel fine to drive home. I just have to think positively and nothing bad will happen! Or, If I visualise perfect health that lump I’ve been ignoring will turn out to be nothing.

In The Uses of Pessimism & the Danger of False Hope, philosopher and fox-hunting enthusiast, Roger Scruton argues against unbridled or, as he puts it, ‘unscrupulous’ optimism, piling many – or most – of the world’s ills at its door. If we always look on the bright side of life then we fall into ‘the best case fallacy’.¹ In Scruton’s words, this leads inexorably to “a kind of addiction to unreality that informs the most destructive forms of optimism: a desire to cross out reality, as the premise from which practical reason begins, and to replace it with a system of compliant illusions.”

This is in no one’s interest. Self help gurus often point to research which reveals that ours brains can't distinguish between imagining positive outcomes and experiencing them, leading to similar neural responses and potential benefits from positive thinking.² But there’s an obvious downside to this: if you can’t tell the difference between thinking positively and achieving a positive outcome, why would you bother to put in the hard work to make your dreams come true?

Those who wax lyrical on the boundless possibilities offered by an exciting future and urge change, progress and the uncritical veneration of the new are recklessly overconfident. They ignore the lessons of the past, the realities of the present as well as the full range of possibilities offered by the future. This isn’t merely a psychological curiosity. As well as distorting our private decisions, unscrupulous optimism shapes public policy, institutional reform and billion dollar procurement programmes.

Consider the Los Angeles Unified School District's ambitious plan in 2013 to provide every student with an iPad, preloaded with Pearson curriculum at a cost of almost $1.3 billion. A classic case of optimistic ambition outpacing reality. Students hacked the iPads to bypass restrictions within days, the Pearson curriculum was incomplete, buggy, or missing key features, teachers were not properly trained to use the devices effectively, and schools lacked the infrastructure to support usage. This was a perfect storm of rushed execution, poor planning, and murky contracts. Needless to say, LAUSD pulled the plug, struggled to ever get compensation from either Apple or Pearson and ended up selling off the iPads at a huge loss.³

How did it all go so wrong? It’s not as if this was the first such disaster. In 2012, one year earlier, plans to hand out laptops to students in Hoboken, New Jersey went pear shaped for some of the exact same reasons: insufficient teacher training and lack of infrastructure.⁴ In 2014, Baltimore County Public Schools launched a district wide one to one device rollout. Costs rose to more than $275 million, the programme became embroiled in procurement controversies, and critics argued that the district had failed to produce convincing evidence of improved educational outcomes.⁵ Remarkably, the exact same thing had already happened in Los Angeles one year before the iPad debacle with the district handing out netbooks with inadequate preparation, poor software and lack of infrastructure and support!⁶ None of these failures was unforeseeable. They were the entirely predictable consequences of assuming technology could solve systemic organisational problems.

The triumph of hope over experience

Although these feel like fairly extreme examples of the best case fallacy, this sort of thing plays out in schools regularly. We routinely launch new policies, spend money on new systems, train teachers in trendy-sounding pedagogical gimmickry and almost never run pilot programmes, or have a clear evaluation plan in place.

Rob Coe’s called his 2013 lecture on improving education The Triumph of Hope Over Experience. In it he observed,

Despite the apparently plausible and widespread belief to the contrary, the evidence that levels of attainment in schools in England have systematically improved over the last 30 years is unconvincing. Much of what is claimed as school improvement is illusory, and many of the most commonly advocated strategies for improvement are not robustly proven to work. Even the claims of school effectiveness research – that we can identify good schools and teachers, and the practices that make them good – seem not to stand up to critical scrutiny. Recent growth of interest in evidence-based practice and policy appears to offer a way forward; but the evidence from past attempts to implement evidence-based approaches is rather disappointing. Overall, an honest and critical appraisal of our experience of trying to improve education is that, despite the best intentions and huge investment, we have failed – so far – to achieve it.

Without such an honest and critical appraisal of our efforts we’re gambling with children’s futures. And the problem with gamblers is that they don’t actually see themselves as risk-takers. They convince themselves that they can beat the odds, that they’ve discovered a fool-proof system to buck the odds. When education leaders estimate the best case and ignore the worst, they might argue the odds look good but, in fact, they are wilfully refusing to acknowledge reason or reality.

Of course, optimism doesn’t have to be unscrupulous. The best leaders have a strong sense of practicality which roots them in the real world. If we recognise the constraints which bound reality and accept that change is difficult and unpredictable we’re less likely to go wrong.

We’re all flawed. We all make poor and irrational decisions, and we’re all hostages to bias and assumptions. To err is human. This is as true of teachers as it is of students; everyone makes mistakes. The solution isn’t to just hope for the best but to be “judiciously pessimistic.”

Judicious pessimism teaches us not to idolise human beings, but to forgive them their faults and to strive in private for their amendments. It teaches us to limit our ambitions in the public sphere, and to keep open the institutions, customs and procedures whereby mistakes are corrected and faults confessed to, rather than aim for some new arrangement in which mistakes are never made. (Scruton p.37)

Instead of seeing only obstacles to be overcome, we would do better to recognise and understand the constraints we face. Some things cannot be done well and shouldn’t be considered. We need to recognise the temptation to assume order will always emerge from a sea of chaos but, in fact, entropy is inevitable; things fall apart. If the dice fall as we wish then we assume it’s as a result of our actions.

Why arguments for coursework are unscrupulously optimistic

Consider the repeated calls to reintroduce coursework into England’s national examinations. The argument is seductive: coursework seems more authentic, allows students to demonstrate a wider range of skills and reduces the pressure of a single exam. All of this may be true. The problem isn’t the principle but the implementation. Once coursework counts towards high stakes qualifications, every incentive shifts. Parents help, tutors rewrite, AI drafts, teachers feel pressure to offer ever more detailed feedback. Schools interpret the rules differently. Moderation becomes increasingly expensive and increasingly unreliable. Before long, the grade reflects not merely a student’s attainment but the resources, expertise and integrity they have access to. This all emerges naturally from thousands of well intentioned people responding rationally to perverse incentives. Every teacher wants their students to succeed. Every parent wants to help. Every school wants good results. Together, these perfectly understandable actions produce a system that is less fair, less reliable and less valid than the one it replaced.

But there’s a deeper, even more uncomfortable truth. The more we rely on teacher judgement, the more we expose ourselves to the biases, expectations and stereotypes that inevitably accompany human judgement. Research has consistently found that teacher assessments can disadvantage students from poorer backgrounds and some ethnic minority groups relative to externally marked examinations.⁷ None of this implies that teachers are especially prejudiced, only that they’re human.

During the pandemic, despite schools acting in good faith, grades drifted sharply upwards. Although some wanted to interpret this as evidence that teachers had suddenly become better judges, a more plausible explanation is that, when external constraints disappeared, the incentives acting on teacher judgement changed. The system behaved predictably. External assessment isn’t perfect, but it provides a degree of protection against the unconscious biases that flourish whenever assessment becomes personal.

Running an optimism audit

Complex systems don’t fail because individuals behave badly but because ordinary people respond predictably to the incentives the system creates. To hope that everyone will exercise restraint is a wager against human nature, and history suggests it’s a very poor bet.