Research School Network: Using evidence properly These four decisions underpin evidence-informed practice

In a recent post, I described how the term evidence-informed practice risks losing its meaning as it becomes more widespread.

Using evidence sounds sensible, but how, precisely, can it add value to our work?

Like many people, I think evidence use involves close consideration of our context, high-quality evidence and professional judgement – but this is too vague. I think evidence can add value to our work by helping us make four decisions. I consider these essentially the mechanisms that lead to evidence-informed practice.

1. Deciding what to do
2. Deciding what to do exactly
3. Deciding how to do things
4. Deciding if things work

Evidence can help us to decide where to focus our effort. The EEF’s Toolkit has hugely influenced these decisions, and the phrase ​‘best bets’ is now widely used when discussing evidence.

The main currency in school is the time of teachers and pupils. Therefore, it is crucial to recognise that evidence can also identify some activities that, on average, had a relatively low impact in the past.

Although this is where people often start with evidence, I think that this is one of the most challenging ways to add value with evidence. The EEF’s implementation process, particularly the explore stage, is very helpful here, but I think the expertise to use it well is spread thinly.

‘It ain’t what you do; it’s the way that you do it’ is another phrase famous amongst people interested in evidence use. I’m not sure I fully appreciated what this meant for a long time. But the popularisation of approaches like retrieval practice has taught me that quality is everything.

School leaders need to define quality. This is necessary to move from a vision to a shared vision, to shared practice. If this is not done well, common issues include superficial compliance, the drift of ideas over time, and difficulties with monitoring and evaluation.

Interestingly, different forms of evidence can help define quality, including observation and reflection, which underpin approaches like Teach Like a Champion and Walkthrus.

Randomised controlled trials alone are rubbish at building theories – though they are still needed to test them. Resources like the EEF’s Toolkit and Guidance Reports helpfully identify areas where teachers should focus their efforts. Still, the insights are rarely granular enough to decide what to do exactly – we need professional judgement.

A striking finding from the work of the EEF and other organisations is that how things are done is just as important as what is done. A striking observation when working with many schools is that some schools can take an idea that does not seem very promising but make it work because they excel at implementation. The reverse is also true.

A common challenge when trying to do things in school is that we have skipped over the first stage. We have not defined – with precision and depth – what quality looks like, which creates all sorts of problems, including that it is impossible to ​‘faithfully adopt’ and ​‘intelligently adapt’.

Consistency is then sometimes pursued as a goal for its own sake. I think this is dangerous without a clear conception of quality and how consistency can add value. It shares some troubling characteristics with fanaticism: when someone redoubles their efforts after they have forgotten their aims.

I described in my previous post that I think evaluation is tremendously challenging to do in schools because the signal-to-noise ratio is so poor: most things we do in school have a small to modest impact, yet many other factors influence outcomes we care about. Therefore, it is exceptionally hard to isolate the effects of specific actions.

Ultimately, if we rely only on best bets, we are gambling. The best way to protect ourselves from a net negative impact of any policy is to find out if it is working as we hope in our schools with our pupils.

I think focusing more on how evidence can add value to our work in schools is essential. Crucially, these different mechanisms of evidence use, or decisions, require different forms of evidence and tools. They also make different assumptions.

If we are serious about using evidence properly in schools, we need to get a lot more interested in the detail of how evidence adds value.