Research School Network: Measure Twice, Cut Once Asking the right questions before running with an intervention

In his first blog for Bradford Research School, Research School Lead Luke Swift details how a research-informed approach can help ensure that you make the right choices in your context. Luke is a class teacher and English coordinator at Dixons Manningham Primary.

I’m sure that many of us have been there: there’s a new intervention presented in a staff meeting and you’re not entirely sure it’s going to work. Well, how can being research-informed help us decide?

I found myself in this situation recently and, working in a research-informed school, there were several conscientious teachers eager to know what kind of evidence base existed for what was being proposed in the meeting. The question was asked and we were assured some research would be provided. In the meantime, I checked my go-to sources for research (knowing these are well-respected, impartial and easy to navigate sources – see bottom of the page for details) which, unusually, all came up blank. I then contacted a couple of experts within the Research Schools network but they couldn’t shed any immediate, further light on the issue this time.

When I received the 6 pieces of research referred to in the meeting supporting the intervention, my next step was to look closely at it and this uncovered some interesting things:

• 2 of the studies were from a non-refereed journal (meaning it has not gone through a formal peer review process for quality assurance);
• 2 of the studies involved only 4 pupils;
• 1 of the studies involved only 5 pupils;
• The other study had a larger sample size (77 pupils) but all were secondary school pupils (not my setting) and the results were actually, on closer inspection, far from conclusive.

This process reminded me of an interesting meeting with the Education Endowment Foundation’s Megan Dixon in which she warned of research ​‘red flags’ outlined by the Institute for Effective Education that are always worth bearing in mind when faced with any research. You can read more in the IEE’s excellent Engaging with Evidence Guide from Jonathan Haslam and Dr Alicia Shaw. The image below is taken from there:

But it is also worth keeping in mind that a lack of robust or reliable evidence doesn’t necessarily mean that something has no chance of working. Dylan Wiliams famously notes, ​“Everything works somewhere; nothing works everywhere,” and the challenge is to try and find out under what conditions something is most likely to work.

The EEF have produced a useful list in their Improving Literacy in KS1 Guidance Report; active ingredients of effective KS1 literacy catch-up programmes are named and encouraged when considering ​‘unproven’ interventions:

I’m not suggesting, of course, that this list is applicable or transferable to all subject areas, age groups and contexts (although some points on the list would be difficult to argue against). But the idea of active ingredients here is potentially a useful one. An active ingredient is, according to encyclopaedia.com, ​“The working element, the component that is designed to affect a desired change”.

Once these active ingredients were identified in the proposed intervention, I tried to cross-reference them on the EEF toolkit to see how effective the various elements of the programme were. This is admittedly not an exact science and perhaps a little crude but it is still better than the alternative in my opinion – as Sir Kevan Collins said at the launch of the Bradford Research school, ​‘using evidence to make decisions is better than using one’s own whim and prejudice’. I found that active ingredients of the intervention that were also covered (or partially covered) in the toolkit
included:

• Feedback +8 months
• Individualised Instruction +3 months
• Mastery learning +5 months
• One to one tuition +5 months

From this then it appears that, despite the 6 questionable studies, the proposed intervention does stand a reasonable chance of having a positive impact on pupils (particularly considering the role of assessment on the teacher’s part in selecting material for this particular intervention).

So here are some steps regarding research that I’d recommend when considering a new intervention at school:

• Ask the source of the intervention about the research supporting it (but be a critical consumer of it);
• Ask a Research School expert (I’ve found they’re always more than happy to engage);
• Do your own reading and searching to check for evaluations – EEF, What Works Clearinghouse or Evidence4Impact, The Chartered College’s EBSCO feature (subscription required);
• Identify the active ingredients of the intervention and assess how effective and relevant those are in your setting.

I appreciate that all these things do take time and energy (extremely precious commodities in schools) but I’d argue that it’s time and energy well spent – as my builder grandfather used to say, ​“Measure twice, cut once!”