Research School Network: Evaluating Impact as an insider A look at how school leaders need to be ware of our insider bias when monitoring and evaluating impact

Insiderness – I mean it sounds like a made-up word doesn’t it? But actually, it is a word, commonly used in research to define an approach in which the researcher is positioned within the settings where they are gathering and analysing information. If you have completed any action research within your organisation, for example when completing a masters degree, you will have come across the term and probably had to write at length about both the pros and cons it.

An insider researcher is assumed to have greater access to information, respondents and data than an external researcher, while also having a greater understanding of the context and nuances of the setting in which the data is gathered. Insider researchers are assumed to have a deeper understanding and clarity of thought in regards to analysing data within their study, allowing them interpret, critique and draw appropriate inferences in greater detail. Insider researchers may often have shared experiences with members of the organisation meaning there is a degree of relationship and trust already built into the data gathering and interpretation process.

However, insiderness also comes with some important caveats. Being an insider researcher does not mean absolute access – just because you are an insider you can’t assume to already posses a comprehensive understanding of what is going on. Insiderness also carries the risk of assuming unbiased perspectives. Familiarity can breed complacency, and as an insider it is important to guard against making predetermined judgements because you think you already know what is happening. Another significant factor to consider when acting as an insider researcher is confirmation bias. This relates to the unconscious tendency to search for or interpret data that supports your existing beliefs or wanted outcomes. This is based in the theory that most people have predilection to readily accept information that supports their existing views and be wary of information that counters this.

At this point you might be thinking ​“well that is all well and good but what has it got to do with me, I am not conducting research in my schools?”, but I would argue that school leaders are in some ways always conducting research. We aren’t necessarily setting up randomised control trials in our schools, but as leaders we have a mandate and moral duty to properly evaluate any strategies, initiatives and interventions ongoing in our schools. One of the challenges for this, is that as leaders in our schools we are insiders. This is unavoidable and we therefore have to be conscious of this when evaluating the success of any strategies. For example, we need to consider that:

• As insiders with responsibility for an initiative (i.e. a new teaching strategy for reading etc) our relationship with our staff may impact our ability to collect data to properly evaluate the success with which this strategy is being implemented. By walking into a room, we may prompt a delivery of the strategy by the teacher which would not have happened otherwise due to the ​“power dynamic” at play.
• Many leaders in schools have been working within that school for a while, they therefore naturally will have strong pre-existing knowledge of teachers, subject/​phase areas etc. and how they ​“usually” take on board new approaches. We need to guard against these pre-existing ideas having an influence on our evaluation
• As leaders we will have spent along time exploring and setting up any change initiative and will want it to be successful – this naturally creates the conditions for confirmation bias to rear its head.

I suppose the questions is how do we work round this? We can’t suddenly have tens of external evaluators roaming our schools evaluating our work, we need to make sure as leaders we are effective evaluators of what our schools are trying to do. We need to be conscious of our insiderness and the pitfalls this brings with it, and use strategies and structures to mitigate these. For example:

• Ensure sufficient time and opportunities are created for staff to reflect on implementation data and feedback.
• Don’t be an island – use your implementation team to help you with monitoring. As part of this be really clear about what you expect to see if your approach has been successful implemented and share this vision with the team, any deviation from this should be noted and investigated further as it may suggest limited reach, acceptance of fidelity.
• Be clear about monitoring fidelity – to what extent are you expecting a new approach to have bene adopted and by when. Be clear how this will involve over time. Fidelity may relate to structural aspects such as the correct number of intervention sessions having been delivered to more complex aspects such as the quality with which a teaching and learning approach is being delivered in the classroom. Thinking about what this will look for both aspect and by when will allow for more effective monitoring.
• Be clear about reach – don’t assume that your work is reaching all corners of your organisation. Pre-assuming that a change in approach has been implemented by some, because they have done successfully before can be dangerous. This may not be the case!
• Don’t be fooled by performance. We talk about students sometimes demonstrating performance rather than learning at the end of lessons – and the same can perhaps be said for staff. Just because we maybe see staff implementing a new approach or any other form of initial success (while worth celebrating) this should not be taken as evidence of a sustained change in approach – monitoring must be scheduled to ensure the change remains live.
• Consider all the sources of evidence that can be used to evaluate your work and identify these prior to beginning your monitoring and evaluation – otherwise it is easy to go looking for the evidence that supports rather than challenges our work.

Ben Crockett

Co-Director Durrington Research School