Luke Sibieta is a Programme Director within the Education, Employment and Evaluation sector at the Institute for Fiscal Studies.
In this piece, he describes a collaboration between Lambeth Council and researchers from the IFS which proved informative for policymakers and of value to researchers.
Local councils in the UK have the potential to be a laboratory for testing policy ideas and interventions. In the US, individual states have frequently trialled different approaches to public service delivery, with successful examples taken up by other states. This process of policy trial and diffusion has been much less common in the UK, at least historically.
The gold standard for evaluating policy tools and interventions is the randomised controlled trial (RCT). RCTs are widely used for medical trials and increasingly used for educational interventions in the UK. However, local councils in the UK (or the national government) have not historically run RCTs to evaluate public policy changes. Designing and running RCTs requires planning, additional survey and measurement costs. Policies need to be randomly varied across citizens. These barriers can seem daunting to a council, and perhaps not part of their culture. External researchers can support and guide councils through confronting each of these challenges.
As an example of how this can be done, researchers from the Institute for Fiscal Studies collaborated with Lambeth Council over a two year period and run a randomised controlled trial to inform the decisions of Lambeth Council and provide wider scientific evidence on how to engage citizens in sustained co-production of public services.
This partnership model produced a successful policy evaluation and along the way we learned a number of valuable lessons.
Building strong and collaborative relationships was absolutely key to success
We started by having a number of conversations with officers from Lambeth Council about the value of robust policy evaluation in the form of RCTs and moved on to discussing the best policy area to focus on. The eventual design of the Street Champions scheme and the associated incentives were a joint design between IFS researchers and Lambeth Council. This meant we had both invested in the partnership and were convinced of the value of the potential findings. The relationships we formed also meant that when things didn’t go completely to plan (as they often do with experiments), we could quickly find a solution.
Taking account of what the council was already doing greatly improved the evaluation
The Street Champions scheme we designed was influence by existing Lambeth schemes to encourage citizen involvement, but extended these to look at the drivers of sustained citizen involvement. We were also able to modify existing local council data collection to improve the evaluation, offering a relatively low-cost way of collecting the right sort of data. For example, we added a small number of extra questions to the annual survey of residents. The evidence of a positive impact of the scheme on perceptions of residents’ local area and social capital were some of the most interesting findings of the evaluation.
RCTs are not necessarily expensive and complicated to deliver
Such concerns are often the main barrier to local councils doing them. However, the main cost of doing RCTs is the collection of new data. In some cases, this can be expensive, such as when doing educational experiments that require individual child assessments. This is not always true. In our case, we collected some new data in the form of repeated observations of litter and street cleanliness for the 170 streets involve, but also made extensive use of existing administrative data and modified existing council data collection methods in small ways to collect new data about the scheme. An RCT is not inherently expensive, its the data you need or want to collect that drives costs.
The design of the scheme and how to interpret the results was where we as academic researchers could add most value. We illustrated to Lambeth how understanding all the ways in which treatment conditions varied across treatment and control groups was key to interpreting and understanding the implications of the results. For example, it was important to have a group who just received a letter inviting individuals to become a Street Champions alongside letters that also included the offer of incentives. This allowed us to isolate the pure impact of the offer and the pure impact of the incentives.
Benefits for both sides
We worked closely with Lambeth Council to develop the design of the experiment and to analyse the policy implications of the data coming out of it. The scheme eventually rolled out was very much informed by the results of the experiment. We as academic researchers also gained a much greater understanding of how local government works and the challenges facing policymakers. We hope that more councils and academics will follow this example.