The updated guidance on Contributions to Knowledge and Understanding (CKU: formerly known as outputs) will be seen as the moment it became clear what REF is.
REF is not about solely, or even mostly, measuring researcher performance. Its primary purpose is to assess how organisations measure research excellence.
It is the release which signals that research may be produced by individuals but it is assessed at an institutional level and the only measure that matters is whether the institution was responsible for supporting the research that led to the output.
2014 Redux
It is worth rehashing how we got here.
REF is the tool Research England and its devolved equivalents use to decide how much QR funding universities will receive. One thing it measures is the research output of universities. The research output of universities are the outputs of the researchers that work there (or a sample of the outputs.)
The question that REF has always grappled with is whether to measure the quality of research or the quality of researchers. The latter would be quite a straightforward exercise and one that has been done in different formats over the years. Get a cross-sample of researchers to submit their best research at a given point in time and then ask a panel to rate its quality.
Depending on the intended policy output the exercise might make every researcher submit some research to ensure a sample is truly representative. It might limit how much any one researcher can submit to ensure a sample is balanced. It might tweak measurements in any number of ways to change what a researcher can submit and when depending on the objectives of the exercise.
The downside of this approach is that it is not an entirely helpful way to understand the quality of university research across an entire institution. It tells you how good researchers are within a specific field, like a Unit of Assessment, but it does not tell us how good the provider is at creating the conditions in which that research takes place. Unless you believe, and it is not an unreasonable belief, that there is no difference between the aggregate of individual research outputs and the overall quality of institutional research.
Individuals and teams
To look at it another way. Jude Bellingham looks very different playing for England than he does Real Madrid. He is still the same footballer with the same skills and same flaws. It is that for Real Madrid he is playing for a team with an ethos of excellence and a history of winning. And for England he is playing for a team that consistently fails to achieve anything of note.
The only fair way to measure England is not to use Jude Bellingham as a proxy of their performance but to measure the performance of the England team over a defined period of time. In other words, to decouple Bellingham’s performance from England’s overall output.
As put in a rather punchy blog by Head of REF Policy Jonathan Piotrowski,
REF 2029 shifts our focus away from the individual and towards the environment where that output was created and how it was supported. This change in perspective is essential for two key reasons: first, to gather the right evidence to inform funding decisions that enable institutions to support more excellent research and second, to fundamentally recognise the huge variety of roles and outputs that contribute to the research ecosystem, including those whose names may not appear as authors and outputs that extend beyond traditional journal publications.
Who does research?
The philosophical questions are whether research is created by researchers, institutions, or both and to what degree. And in a complex system involving teams of researchers, businesses, and institutions, whether it is any easier or accurate to ascribe outputs to researchers than it is to institutions. The policy implication is that providers should be less concerned about who is doing research but the conditions in which research occurs. The upshot is that the research labour market will become less dynamic, there is less incentive to appoint people as they are “REFable”, which will have both winners and losers.
The mechanism for decoupling in REF 2029 is to remove the link between staff and their outputs. The new guidance sets out precisely how this decoupling process will work.
There will be no staff details submitted and outputs will not be submitted linked to a specific author. Instead, outputs are submitted to a Unit of Assessment. This is not a new idea. The 2016 review of the REF (known as the Stern Review) recommended that
The non-portability of outputs when an academic moves institution should be helpful to all institutions including smaller institutions with strong teams in particular areas which have previously been potential targets for ‘poaching’.
However, it is worth emphasising that this is an enormous change from previous practice. In REF 2014 the whole output was captured by whichever institution a researcher was at, at the REF census date. In REF 2021 if a researcher moved between institutions the output was captured by both. In REF 2029 the output will be captured by the institution where there is a “substantive link.”
Substantive links
A substantive link will usually be demonstrated by employment of a period of 12 months at least 0.2 FTE equivalent. The staff member does not have to be at the provider at the point the output is submitted. Other indicators may include
evidence of internal research support (for example, funding for research materials, technical or research support, conference attendance) evidence of work in progress presentations (internally and externally) evidence of an external grant to support a relevant program of research.
In effect, this means that the link between researchers and REF is that their research took place in a specific institution, but it is ultimately the institution that is being assessed. The thing that is being assessed is the relationship between the research environment and the creation of the output. Not the relationship between the output and the researcher.
As the focus of assessment shifts so do the rules on what can or cannot be submitted. As we know from previous guidance there is no maximum or minimum submissions from staff members. There may be some researchers at, or who were at, a provider who find their work appears in an institution’s submissions a number of times, and maybe even across disciplines (there will no be now no inter-disciplinary flags but an output may be submitted to more than one UOA and receive different scores.)
The obvious challenge here is that while providers should submit representative outputs the overriding temptation will be to submit what they believe to be their “best” and then work backwards to justify why it is representative. The REF team have anticipated this problem and the representativeness of a submission will be assessed through the disciplinary led evidence statements. The full guidance on what these contain is yet to be released but we know that
The important issues of research diversity, demographics and career stages will be assessed as part of the wider disciplinary level evidence statements
Research England’s position is that aligning outputs to where they are created, not who creates them is a better way to measure institutional research performance. This should also end the incentive for universities to recruit researchers and in doing so capture their REF output. The thinking is that this favours the larger universities that can afford to poach research staff.
Debates had and debates to come
In a previous piece for Wonkhe Maria Delgado, Nandini Das, and Miles Padgett made the case that portability is key to fairness in REF. The opposite argument that is being put forward by Research England. Maria, Nandini, and Miles made the case that whether we like it or not one of the ways in which academics secure better career prospects is by improving the REF performance of a provider’s UOA. Research England makes the case that
The core motivation is to minimise the REFs ability to exert undue influence on people’s careers. To achieve this, institutional funding (remember, QR funding does not track to individuals or departments) should follow the institutions that have genuinely provided and invested in the environment in which research is successful. Environments that recognise the collaborative nature of research and the diverse roles involved, rather than simply rewarding institutions positioned to recruit researchers to get reward for their past output.
It is possible that both arguments may be right. If outputs are tied to institutions the incentive for institutions who want to do well in REF is to capture a greater number of high quality outputs to include in their submission. The way to do this is to have more researchers supported to do high quality work. On the other hand, at an individual level and in a time of financial crisis for the sector, there are likely some researchers who benefit from being able to take their research output with them when they move institutions.
In the comments of our initial portability piece it was flagged that researchers’ work could form part of an assessment where they had no relationship with the provider. This feels particularly egregious if they have been made redundant as part of wider cost saving. The message being that the research output is high quality but nonetheless it is necessary to remove your post. The REF team have considered this and
Outputs where the substantive link occurred before the submitted output was made publicly available, will not be eligible for submission where the author was subject to compulsory redundancy.
The guidance explains that there may be times where there is a substantive relationship but the research has not yet been published. On the face of it this seems a sensible compromise but if the logic is that a provider is the place where research outputs are created it seems contradictory (albeit kinder) to then limit the conditions through which that work can be assessed. It is possible there will be some outputs which were in the process of being published but not yet assessed which would fall into this clause.
The guidance confirms a direction of travel that was established as far back as REF 2021 and made clear in the guidance so far for REF 2029. While the debate on who should be assessed in which circumstances continues the wider concern for many will be that there is still significant guidance outstanding, particularly on People Culture and Environment, and the submission window for REF closes in 30 months from now.
A direction has been set. The sector needs to know the precise rules they are playing by if they are going to go along with it. There is undoubtedly a lot of good will around measuring research environments, culture, and the ways in which outputs are created more comprehensively. That good will, will evaporate if guidance is not timely, clear, or complete.
