The dangers of optimism: The best case fallacy

Why optimism is often unscrupulous and positive thinking makes things worse

Optimism, n. The doctrine or belief that everything is beautiful, including what is ugly.

Ambrose Bierce, The Devil’s Dictionary

The world - and particularly the internet - is full of encouragement for us to think positively. If only we were positiive enough, all the good things will happen.

There are even people who tell us the reverse: that all the bad things that happen to us are the fault of our own negativity. This is, I’m afraid to say, bullshit. It’s also potentially dangerous: I’ve just had a few pints at the pub and I feel fine to drive home. I just have to think positively and nothing bad will happen!

In The Uses of Pessimism & the Danger of False Hope, philosopher and fox-hunting enthusiast, Roger Scruton argues against unbridled or, as he puts it, ‘unscrupulous’ optimism, piling many – or most – of the world’s ills at its door. If we always look on the bright side of life then we fall into ‘the best case fallacy’. In Scruton’s words, this leads inexorably to “a kind of addiction to unreality that informs the most destructive forms of optimism: a desire to cross out reality, as the premise from which practical reason begins, and to replace it with a system of compliant illusions.”

This is in no one’s interest. Self help gurus often point to research which reveals that ours brains can't distinguish between imagining positive outcomes and experiencing them, leading to similar neural responses and potential benefits from positive thinking. But there’s an obvious downside to this: if you can’t tell the diffeence between thinking positively and acheiving a positive outcome, why would you bother to put in the hard work to make your dreams come true?

Those who wax lyrical on the boundless possibilities offered by an exciting future and urge change, progress and the uncritical veneration of the new ignore both the lessons of the past, the realities of the present and the full range of possibilities offered by the future. This all sounds uncomfortably familiar. How many of us in education spurn the past, decry the present and urge that we embrace something - anything - new.

Consider the Los Angeles Unified School District's ambitious plan in 2013 to provide every student with an iPad, preloaded with Pearson curriculum at a cost of almost $1.3 billion. A classic case of optimistic ambition outpacing reality. Students hacked the iPads to bypass restrictions within days, the Pearson curriculum was incomplete, buggy, or missing key features, teachers were not properly trained to use the devices effectively, and schools lacked the infrastructure to support usage. This was a perfect storm of rushed execution, poor planning, and murky contracts. Needless to say, LAUSD pulled the plug, struggled to ever get compensation from either Apple or Pearson and ended up selling off the iPads at a huge loss.

How did it all go so wrong? It’s not as if this was the first such disaster. In 2012, one year earlier, plans to hand out laptops to students in Hoboken, New Jersey went pear shaped for some of the exact same reasons: insufficient teacher training and lack of infrastructure. In 2014, Baltimore public school system also launched a 1-1 device roll out which ran into ballooning costs and garnered no discernible positive results. Remarkably, the same thing had already happened in Los Angeles one year before the iPad debacle with the district handing out netbooks with inadequate preparation, poor software and lack of infastructure and support! This is the triumph of hope over experience.

These are extreme examples of the best case fallacy but this sort of thing plays out in schools regularly. We routinely launch new policies, spend money on new systems, train teachers in trendy-sounding pedagogical gimmickry and almost never run pilot programmes, or have a clear evaluation plan in place.

Rob Coe’s called his 2013 lecture on improving education The Triumph of Hope Over Experience. In it he observed,

Despite the apparently plausible and widespread belief to the contrary, the evidence that levels of attainment in schools in England have systematically improved over the last 30 years is unconvincing. Much of what is claimed as school improvement is illusory, and many of the most commonly advocated strategies for improvement are not robustly proven to work. Even the claims of school effectiveness research – that we can identify good schools and teachers, and the practices that make them good – seem not to stand up to critical scrutiny. Recent growth of interest in evidence-based practice and policy appears to offer a way forward; but the evidence from past attempts to implement evidence-based approaches is rather disappointing. Overall, an honest and critical appraisal of our experience of trying to improve education is that, despite the best intentions and huge investment, we have failed – so far – to achieve it.

Without such an honest and critical appraisal of our efforts we are gambling with children’s futures. The problem with gamblers is that they don’t actually see themselves as risk-takers. They convince themselves that they will beat the odds. When education leaders estimate the best case and ignore the worst, they might argue the odds look good but, in fact, they are refusing to acknowledge reason or reality.

Of course, optimism does not have to be unscrupulous. The best leaders have a strong sense of practicality which roots them in the real world. If we recognise the constraints which bound reality and accept that change is difficult and unpredictable we are less likely to go wrong.

We’re all flawed. We all make poor and irrational decisions, and we’re all hostages to bias and assumptions. To err is human. This is as true of teachers as it is of students; we all make mistakes. The solution is to be judiciously pessimistic.

Judicious pessimism teaches us not to idolise human beings, but to forgive them their faults and to strive in private for their amendments. It teaches us to limit our ambitions in the public sphere, and to keep open the institutions, customs and procedures whereby mistakes are corrected and faults confessed to, rather than aim for some new arrangement in which mistakes are never made. (Scruton p.37)

Instead of seeing only obstacles to be overcome, we would do better to recognise and understand the constraints we face. Some things we cannot do and should not consider. We need to recognise the temptation to assume order will always emerge from a sea of chaos but, in fact, entropy is inevitable; things fall apart. If the dice fall as we wish then we assume it’s as a result of our actions.

We forget or ignore this at our peril. By all means imagine the best case, but temper the desire to tamper with a dose of the worst case. Maybe before making any kind of decision, we should be made to ask ourselves these questions:

• Have you considered the real root cause of the problem you’re trying to solve?

• Have you considered other possible reasons for the problem?

• Have you sought out sources and evidence which contradict your beliefs?

• Have you allowed for dissenting opinions to be voiced and considered?

• Have you considered the weight of time, resources and credibility you or others have already sunk into this course of action?

• How might groupthink and social proof be influencing your decisions?

• Have you encouraged others to criticise and suggest problems with your plans?

• How far is your decision based on your opinion of the individuals concerned?

• To what extent are your decisions anchored by possibly irrelevant information?

• Do you really understand the data you’re using to inform decision-making?

• What perverse incentives might you be creating?

• Certainty often blinds us to alternatives. How confident are you that your decision is correct?

• What would be the consequences of not taking this course of action?

• Have any other schools tried this course of action? How many were still doing it three years later? What were the results?

• Who else could you ask to spot the biases in your thinking?

• Have you put an evaluation plan in place before launching your new idea?

• And, whenever possible, run a pre-mortem to work out what could go wrong in advance.

Stuart Kime wrote a great article in Schools Week on avoiding the best case fallacy a few years ago: How to avoid project failures in the classroom

Thanks to Beth GG who gave a great talk on negative thinking at researchED Scandinavia and introuduced me to Oliver Burkeman’s BBC podcast on The Power of Negative Thinking