We need to rethink ‘impact’ – the findings from the Longitudinal Studies Strategic Review

by Rebecca Fairbairn

In 2016 the ESRC set out to review our vast Longitudinal Studies portfolio with an aim to fully understand the future scientific needs for survey data gathered on people throughout their lives. We did this to ensure the development of meaningful, robust and impactful research resources: resources that would be as relevant to society in the future as those we currently support are now.

The ESRC’s portfolio has data spanning 60 years with studies including the 1958 National Child Development Study, the 1970 British Cohort Study, the Millennium Cohort Study (all hosted by the Centre for Longitudinal Studies) and Understanding Society. Although the UK is world-leading in this sector and impacts from these studies relate to increased understanding of genetic, health, educational, social and economic dynamics that influence individuals’ lives, it was essential that we took a step back so that we could be confident our future investment plans are well designed for future research needs.

Fast forward into 2018 and following a huge amount of hard work, for which I am truly grateful, from our independent panel (PDF), our steering group (PDF) and my team in the ESRC office, the review is now complete.

LS Review cover

The key outcomes

  • The review has demonstrated the importance of ESRC’s investment in longitudinal data and recommends our investment continues. Historical investment in longitudinal studies gives the UK research base and policy communities a competitive advantage in understanding life course trajectories and the impact on individuals of circumstances, policies and other social changes in the UK. This understanding cannot be replaced supplanted by “big” or administrative data, we need survey data to understand experiences, perceptions and attitudes and to connect early life experience and biomedical data with later life outcomes.
  • There is genuine interest globally in ESRC’s support for longitudinal studies and investments that work to maximise their use. This is clear in the international usage of our studies (some 20% of CLS and 12% of Understanding Society downloads), from the enthusiasm and commitment of our international review panel, and from international co-funding.
  • Something isn’t working in the way ESRC, our studies and those using our longitudinal resources capture and demonstrate impact. I have been most struck by the panel’s findings that

“although numerous studies have been published from ESRC-funded longitudinal studies, the review panel has found it difficult to find specific evidence of the policy impacts from these investments… Impacts from these ESRC longitudinal data investments undoubtedly exist but are hard to pinpoint and quantify… If left unchecked, a lack of evidence is likely to undermine the case for future investments, given the general principle that research funding in the UK should be founded on an evidence-based scientific case….”

 What the longitudinal community needs to do now

Our community is highly engaged – I know that from the sheer volume of responses received during our initial survey – and I know the data in the studies are frequently and widely used. I also know that in the office we are always busy looking for and reporting impact stories. And we have some absolutely cracking examples, from the discovery that smoking in pregnancy harms babies to the development of the new Automatic Enrolment in Workplace Pensions scheme. More generally the existence of longitudinal data from a range of studies, which can be explored in different ways by many researchers, has enlightened our knowledge and understanding of complex social phenomena – often they are the source of what becomes common knowledge – and the policy interventions designed to tackle these. The dynamic nature of poverty is a good example of this: it’s not so long ago that we thought people were born into poverty and would die there; we now understand that being in poverty can be transitory, sometimes repeatedly, but often not continuous.

generations-462134_1920

However, something isn’t quite working.

The methods and issues section at the end of Annex C (Evidence of Influencing Policy) to the report (PDF) neatly outlines the challenges the panel faced during the review year. To paraphrase:

  • Demonstrating impact is particularly difficult for the social sciences and humanities where impacts tend to relate to processes and relationships rather than products, and tend to build over time and across studies rather than emerging from one project or researcher. Capturing impact in this context is complex. Researchers need to commit to properly tracking impact from their work, and as funders, we need to adequately resource this and work together to make sure that information systems are consistently linked.
  • Demonstrating impact from social science is further complicated as research into complex social problems inevitably runs into highly ideological and politicised policy debates.
    (Think about the grammar school debate, for example: research using our longitudinal studies robustly demonstrated that grammar schools do not promote social mobility).
    We, the social science community, need to report what social science tells us and respond with confidence in our science when our findings are reported as being politically based.
  • Demonstrating impact is even more tricky for the data resources we support that are using ESRC’s data policy!
    I am proud of ESRC’s data policy. Its first principle is that “Publicly-funded research data are a public good, produced in the public interest, which shall be made openly available and accessible with as few restrictions as possible…” But this gives us a bit of a problem when we’re trying to capture the impact of our data resources. Like the NHS, our data is free at the point of delivery, but not actually free. Unlike the NHS users need only to “access” (download) a dataset once to benefit from it throughout their career. And sadly elections are not fought on our social science data infrastructure.
    We need to find a way forward where we can continue to make our data readily available while ensuring users are reporting on use and impact. If we don’t close this feedback-loop, we won’t be able to protect investment in the data.

What is impact exactly?

So, we need to re-think what we mean by impact.

Somehow we need to be able to capture data on the impact of relationships and conversations, on impacts of the counterfactual, and the policies avoided as a result of social science evidence.

We need “better methods for tracking the use of ESRC’s longitudinal data.” We are talking with the British Academy about this now and look forward to the work they are planning; it will be truly fascinating.

What next?

Now that the review report has just been published you may think that our work has finished. But really, for the team in ESRC, we are only now at the beginning of our journey.

The recommendations provided by our independent review panel are wide-ranging, covering our existing investments as well as exciting new ideas for focus in relation to administrative data, a new birth cohort and work building on the success of CLOSER to maximize comparability between, and discoverability, and use of longitudinal data.

These recommendations speak to the future needs of social science research, but, like our existing data and infrastructure activities, they are also important elements of the UK’s world leading data-driven research endeavour, and will be vital for tackling the complex social challenges in the future. We are working hard on our impact conundrum, on ensuring our social science data infrastructure needs are captured within the UKRI Roadmap, and on continuing to support our existing investments while we develop new plans that build on the recommendations of the review.


Rebecca Fairbairn 150

Rebecca Fairbairn is joint head of Data and Infrastructure for ESRC. She is responsible for ESRC’s longstanding, world class, portfolio of longitudinal studies and the shape and sustainability of that portfolio into the future. She moved to this role in spring 2016 following three years as head of Knowledge Exchange, where she led the redevelopment of policy for knowledge exchange and impact, transforming the ESRC’s approach to one in which this activity is embedded across the whole of our remit.

Read the full Longitudinal Studies Review 2017 report on the ESRC website.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s