Last week, the EEF released findings from a major evaluation into setting in mathematics. The headlines that followed revealed the different ways in which educational evidence can be interpreted and presented:
Setting in maths leads to better outcomes, EEF study concludes Teaching in classes grouped by ability does not hamper progress of less able pupils, study finds Why mixed-ability teaching is down, but not out Another progressive myth has been demolished
When the same study can produce such different headlines, how do we identify the signal from the noise?
Start with the summary
You can read the summary on the EEF website here. This captures key findings and much of the detail of the study. These are aspects that often get lost in headlines.
Students in schools using mixed attainment grouping made less progress in mathematics compared to students in schools using setting.
Students eligible for free school meals made similar progress in both groups. Low-prior attaining students made similar progress in both groups.
High-prior attaining students made, on average, two fewer months’ progress in mixed attainment grouping compared to students in schools using setting.
Schools’ teaching practices are broadly similar across schools. Schools using setting appear to be challenging high-prior attaining students.
Look at the difference in the headline for this summary:
The Student Grouping Study explored the difference in attainment outcomes for pupils taught maths in mixed attainment groups compared to setting.
Now compare to the EEF news item:
High-attaining maths pupils make more progress when grouped by attainment, EEF and UCL research finds
Neither headline is inaccurate, but each frames the study differently. The project summary describes the focus of the research; the press release foregrounds the most striking outcome. Both shape how readers are likely to interpret the findings before they engage with the study itself.
Go to the source
If you want to fully understand what this study showed, or you’re planning to make changes based on findings, you probably do need to read the whole study. However, it’s eighty-six pages long, and that’s a lot for time-poor teachers. If you wanted to do a targeted reading, I’d look at:
Executive summary – key findings communicated succinctly. Introduction – this sets out the evidence base and rationale for the study. This can be helpful in placing the individual study in the wider evidence context. This section also articulates what the researchers were trying to find out. Conclusion - this is more nuanced than the evidence summary. You will find more of the grey areas, caveats and implications for further research here.
Reading these sections together helps build a fuller picture of what the study found, and how the findings should be interpreted and applied. And a similar approach would work for most evaluations.
The implementation picture
If you are thinking about switching from mixed attainment grouping to sets, or under pressure to do so, the implementation and process evaluation is useful, because it looks at the ‘how’ as well as the final outcomes. In this study, this was made up of ‘surveys in all participating schools, of heads of mathematics, mathematics teachers, school leaders, and students.’ 12 schools were also randomly selected for in-depth case studies, which included observations of lessons.
This section contains some findings that are easily overlooked in broader discussions of the study:
Reasons for setting – heads of maths using mixed attainment grouping were more likely to talk about equity; those using setting were more likely to focus on pupil learning and the ease of teaching in sets.
The majority of heads of maths in setting schools had chosen that approach themselves, whereas mixed attainment grouping was often decided by SLT or the MAT. (Some heads of maths did not agree with the decision.)
Nearly 15,000 pupils were surveyed, and experiences were broadly similar across both approaches. Classroom observations suggested pedagogy was often more similar than different.
Mixed attainment teaching was felt to resemble ‘bottom set’ teaching more than ‘top set’ teaching, with concerns that higher attaining pupils were not sufficiently challenged.
These aspects of the study receive far less attention than the headline outcomes.
Finding informed commentary
We cannot always dive deeper into every evaluation – the EEF published five last week and this is the only one I’ve had the chance to properly read – so we sometimes have to rely on ‘trusted sources’.
In this TES article, Jeremy Hodgen, Professor of Mathematics Education, and Becky Taylor, Professorial Research Fellow, both at the UCL Institute of Education, discuss the findings of the study.
Becky Allen’s article goes beyond the study itself, but is a nuanced exploration of the ‘debate’.
Of course, no commentary is completely neutral. We all bring assumptions, experiences, and values to the way we interpret evidence.
Educational research rarely settles debates as neatly as headlines suggest, which makes identifying the signal from the noise all the more important.
