Impact evaluations can show what factors are affecting your outcomes and how to improve those outcomes. This post looks at Get the Data’s impact evaluation of changes in the youth justice system in England and Wales.
You go to a friend’s house for dinner, and she serves the best chilli you’ve ever had. It’s so good you ask her for the recipe. She doesn’t have it written down because it’s an old family tradition. Still, she quickly jots down a list of ingredients and estimates the quantities she uses: about half a cup of coriander, not too much capsicum, and as much chilli as you can handle. When you come to make the chilli next week, it’s good, but it’s not good as your friend’s. You need to change some of the quantities, but there are ten ingredients. Which ones do you change? And by how much?
When multiple factors affect an outcome, it can be difficult to know what effect they are all having. Changing individual factors, one after another, can be time-consuming and resource intensive and can fail to account for the effects of aggregation. As with your friend’s recipe, you know what the outcome you want is, but you’re unsure what to change to achieve it. This is when you need an impact evaluation.
An impact evaluation sounds very dramatic, but it’s really just a way of measuring the effect of change. It’s a method for discovering what factors affect outcomes, but more importantly, it can show how those factors affect outcomes.
Impact evaluations are not just about measuring; they can also prove. And by proving what works, organisations can continue to make positive changes that will improve outcomes without the need for guesswork. An impact evaluation is an excellent tool for improving outcomes.
Measuring the Impact of Approaches to Youth Justice
Let’s look at a case study to get a better idea of what an impact evaluation is.
In 2021 the Ministry of Justice of the UK Government (MoJ) commissioned Get the Data to assess the impact of its new AssetPlus assessment tool on youth offending outcomes.
In recent years the UK has shifted towards a desistance model of youth offending, where the desired outcome is reducing individual offending behaviour. In 2014, as part of this change of approach, Youth Offending Teams (YOTs) in England and Wales began to use a new tool, AssetPlus, to assess the needs of young people engaged with the youth justice system.
Our evaluation measured various indicators to see if the use of AssetPlus had led to improved outcomes. Our initial feasibility study identified eight outcome measures for which impact could be estimated (out of 18 that the client had suggested). These outcome measures included the time taken to complete assessments, levels of engagement with YOTs by young people, and changes in offending behaviour.
Because of the different ways in which outcomes can be measured, we used various methods to conduct the evaluation. One of these methods was a historical control group.
Historical control evaluations work on a similar principle to a scientific experiment in which the factor being investigated is measured against a control. For this evaluation, the control group contained children who had started their youth offending orders before the introduction of AssetPlus. The control group members were matched to the members of the intervention group by means of propensity score matching – a way of ensuring that the groups were composed of similar individuals. By matching the young people in each group in this way, the team were able to control for factors outside of AssetPlus that may have impacted outcomes. Using this analysis method, we can show how the use of AssetPlus was changing outcomes.
Our impact evaluation proved that AssetPlus positively increased compliance with youth offending orders. Based on this evaluation, we made recommendations to the MoJ to improve outcomes further through training, targeted assessment and data monitoring.
An Impact evaluation is a tool which allows you to see what’s working and why it’s working. Proving what causes outcomes to change means your organisation can concentrate on improving outcomes immediately rather than relying on inefficient trial-and-error approaches.
Get the Data’s Prove package is designed to show you how to improve outcomes. Using your data and our expertise, we can prove what works and help your organisation improve its outcomes. Contact Alan or Jack today to find out how we can help you.