Research School Network: Becoming a Critical Consumer of Research

I recently attended the Research Schools’ Network regional meeting (in Grantham!) where the main focus was on ​“Becoming a Critical Consumer of Research”. We looked at how to assess the applicability of different evidence sources (you’ll need more certainly about the validity of information if you’re moving your whole family to live in America than if you’re simply planning a weekend away in St Ives … for example), evaluating the credibility of evidence sources and effectively responding to a piece of research (the ​“so what?” – now that I’ve learnt something, what am I going to do about it … if anything!)

At every stage when trying to decide how ​“good” a piece of research is (and whether it’s relevant to me/​my school/​my department/​to answer my specific questions in my specific circumstances), what might make me nervous?

As Megan Dixon (Director of Research and Development at the Aspire Teaching School Alliance) said, when evaluating research it’s always wise to:

“employ a degree of professional scepticism”

There are a number of alarm bells that might ring when I study an article or report and ask Who, Where, How, What, When?

Who is the author? Is the author qualified to comment? Do they have experience in the field? Does the author have any potential conflicts of interest?

Where is this published? Is the publication credible? Has the article been peer reviewed?

Why has it been written? Does the author have biases or vested interests, or is the article shared on a commercial website that benefits from the intervention or approach?

How has the research been conducted? Beware of research methods using small or biased samples, dissimilar control and intervention groups or measures testing things that only the intervention group have learnt

What could catch you out in reviews of previous research?

• Authors only reporting studies which support their views and ignoring any other evidence (we’re all guilty of this one when it comes to the positive view we might take of media reports saying wine/​coffee/​beer have a positive health benefits!)
• Overcomplicated (or made up) scientific language.
• Applying findings to situations or people beyond the original research.
• Conclusions which most experts in the field would disagree with

What are the pitfalls you should watch out for when people present data and conclusions?

• Is the sample representative of the population you are interested in?
• Is the context relevant to your school?
• People using correlations to conclude that A causes B
• Check the axes of graphs are not misleading
• Only reporting findings that support the author’s position.

This is all good advice, but we’re all busy teachers with impossible demands on our time (and maybe a desire to carve out just enough time to go and see our youngest child in their nativity play this year, at last …)

So how do the EEF aim to help?

Firstly, they commission literature reviews to assess all the research that’s already been done on an area of interest and produce summary reports. Reviews published by the EEF in the last couple of years include improving English and maths outcomes for 16 to 18 year olds (2016), and careers education (2016).

Secondly, they manage further, specific research … and then very importantly, boil down all the information and present it as toolkits and implementation guidance reports – and all that information comes annotated with clear ​‘evidence strength’ indicators which clearly identifies whether research is based on limited, moderate or extensive evidence.

… but even so, we’re all still too busy to really take in the reports that have been prepared for us.

Enter, the Research Schools

Our remit as the West Somerset Opportunity Area Research School is to guide you through those pieces of research that might be applicable to you and your circumstances … and point you towards the information about implementation methods that have proven successful. Because of course even the most solid piece of Evidenced Based Research only has a positive impact as successful as its implementation.

And the final word on the subject comes from Jonathan Haslam of the IEE:

“we have to be cautious about being too certain”

We need to keep our brains engaged at all times – this is not only about how reliable a body of research is, it’s also about how these research findings apply to our students in our school and in our particular circumstances. Good luck out there, folks!