Research School Network: Evaluating what works Damian Haigh, ELE for Aspirer, explores how we can begin to critically evaluate evidence to inform our decisions.

IS SCRATCHING GOOD FOR YOU?

The debate about fake news is nothing new. Roughly 2400 years ago in Protagoras, Plato wrote:

​“Knowledge is the food of the soul; and we must take care, my friend, that the Sophist does not deceive us when he praises what he sells, like the dealers wholesale or retail who sell the food of the body; for they praise indiscriminately all their goods, without knowing what are really beneficial or hurtful.”

A great deal of ​“knowledge” about how we should teach is presented to us as ​“best practice” but, as Dylan Wiliam often says, ​“everything works somewhere but nothing works everywhere”. If we try to adopt every appealing teaching initiative, we will soon be carrying an unbearable burden and our students will be no better off. The trick is knowing which things we can justly expect to be nutritious and beneficial for our students and our colleagues, and which things we can safely leave on the plate.

It is probably unfair to compare fake news with less effective initiatives in teaching. Teachers who bring forward new ideas are rarely doing it for self-aggrandisement, politics or any other questionable reason: they are trying to make learning better for children and sharing their ideas out of altruism and generosity. However, like the colleagues who generously share their home baking at work, their kindliness can lead to us consuming things that have an unintended effect on our health.

One way to check whether an idea is likely to be genuinely ​“nutritious” rather than just delicious and high in calories is to stop and consider how plausible it is given what you already know. I am currently enjoying learning to code in R and Python for data science purposes. I’ve always thought that learning computer programming cannot be anything but healthy for the learning of mathematics and, particularly, of algebra. When I heard that the popular Scratch programming language had been adopted by UCL Institute of Education’s London Knowledge Lab as a way of helping learners in mathematics it struck me that this cannot fail to be a good (and mathematically nutritious) idea. Highly plausible and, surely, it’s worth a little try?

Fortunately for the calorie conscious among us, a large number of schools have given it a taste already and done so in rigorous way as part of a formal evaluation conducted by Sheffield Hallam University’s Institute of Education.

https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/scratch-maths

Sadly, it seems this treat is not beneficial in the ways we might expect. Based on a sample of nearly 6000 learners the impact on Key Stage 2 maths attainment is zero, with a very small, but not statistically significant, positive effect for ​“ever 6” FSM students. There is some evidence of a benefit for the learners’ computational thinking, but this does not seem to have transferred to their performance in the end of Key Stage 2 assessment.

At least we now know and can direct our efforts elsewhere. Professional development for teachers on the delivery of the computing curriculum is clearly a good thing, particularly if time and money are not limited, but we should not expect the investment to pay off in Key Stage 2 maths scores.

I’m left with a couple of questions about this:

1) Scratch is an intuitive tool for helping students to make a start on programming and develop understanding of the structure of certain types of programming languages, but it would not be my first choice for developing learners’ algebraic concepts. Are there benefits to be had from other, more formal, languages for this purpose? It appears from this rather old but well cited paper that there may be: https://link.springer.com/article/10.1007/BF00555720

2) Key Stage 2 scores are a measure of mathematical ability with only a small algebraic content: is it possible that experience of Scratch projects may have latent benefits for learners of mathematics, and that by the time they are required to solve problems with algebra on a regular basis those with coding experience may find they have an advantage in their thinking? I would be interested to see the data for these 6000 students’ GCSE grades in a few years’ time. It seems unlikely that it will be considered worth investing in the expense of doing this given the lack of impact seen at this stage, but the follow up need only be a quick query to the National Pupil Database. I’ll put it in my diary for 2023; by then my data science skills might be sufficiently up to scratch.

Damian Haigh, ELE Aspirer Research School, January 2019