The publication of the Office for Students’ annual data set on degree classifications and grade inflation was initially scheduled for October of last year.
It was delayed until now to enable further data checking – a pause that caused many data-literate observers to speculate that perhaps the venerable and much-criticised OfS algorithm (which compares the classification of degrees awarded to the perfect year that was 2010-11; controlling for age, entry qualifications, and subject of study only) might be in for an overhaul.
This algorithm has generated results in the past that suggests that more than half of the classifications actually awarded to undergraduates were “unexplained” – the current number is just under 40 per cent.
So, either four in ten degrees awarded in UK higher education are problematic – or a very simplistic algorithm isn’t actually very good and needs fixing.
Occam’s razor
So we thought OfS would take the extra weeks to rethink the algorithm. This has not happened.
Instead, we get a more nuanced take on what is visible in this collection, which is worth quoting in full:
The term ‘unexplained’ in this context means that changes in the characteristics of the graduating cohort included in our modelling cannot explain statistically the changes in attainment over the period.
We are not seeking to understand what other factors might be driving the observed changes. We acknowledge that elements such as improvements in teaching quality could account for them. Our modelling cannot account for increases in degree awarding as a result of changes made in response to the pandemic. Neither can it account for entry requirements such as performance in an audition or the submission of a portfolio, as entry qualifications are limited to standard A-levels, BTECs and direct equivalents.
Similarly, it cannot account for changes in entry qualifications as a result of the teacher-assessed grading necessitated during the pandemic. For this reason, we also classify these changes as ‘unexplained’.
In reading this very welcome clarification you may want to think back to November’s OfS intervention on these topics. After investigating three providers (starting in 2022) England’s regulator appeared to decide that the problem was degree algorithms.
A degree algorithm is the mechanism used by providers to calculate degree classifications from a set of module marks achieved by a student during their undergraduate study. This is a particularly British problem – in most systems globally a grade point average backed by a full transcript is far more important than any classification offered.
In the three investigations OfS conducted it identified two particular aspect of degree algorithms – awarding a student the best result from multiple algorithms, and discounting credit with the lowest marks – that it was unsure were compatible with the requirements of registration condition B4 (which deals, in part with the “credibility” of degrees awarded).
This was a new departure for a regulator that had previously been content to use words like “unexplained” to cast suspicion on academic standards more generally. The fact that it found three providers at risk of breaching B4 despite the absence of any current practice that would be in breach of B4 merely served as an indication that the game has changed.
The hardest degree
We get the usual data release alongside the report. Here’s a plot showing the percentage point difference between the actual grades awarded and the grades modelled by the algorithm (the so-called “unexplained” awards) – with the number of graduates shown by the thin grey lines. Filters allow you to look just at first class honours or first and upper second degrees, choose the year you are interested in (the most recent, 2023-24, is the default), and to choose a minimum number of graduates at a provider for display (the default is 500).
Mousing over one of these marks shows, in the chart at the bottom – the actual (orange) awards plotted alongside the modelled (blue) awards.
Top of the charts for “unexplained” first and upper second awards we find Goldsmiths, East London, and Bradford. With the exception of Goldsmiths’ all recorded a slight drop in the actual (observed) award of undergraduate degrees with these classifications each year.
Like many providers at the top end of this chart, these institutions take pride in serving under-represented and non-traditional applicants to higher education – and they are very good at what they do. Goldsmiths’ is a large arts-focused institution, with admissions determined by portfolio in many cases. East London and Bradford are vocationally-focused providers with strong employer links, serving a local non-traditional population.
East London and Bradford award a far lower proportion of first class and upper second degrees than – for example – Durham, Bath, or Bristol. In any meaningful, student-facing, interpretation of this phenomenon it is “easier” to get a good degree at a selective provider like that than at one more focused on serving the whole community. The hardest university to get a good degree at is Buckinghamshire New University – less than half of those who completed their course in 2023-24 achieved a first or upper second.
It’s perhaps easier to see this phenomenon on a scatter plot showing both observed and modelled awards.
There is a neat split by provider type – every Russell Group university awards more than 80 per cent of graduates a first or upper second, while only a handful (Bath, Loughborough, Lancaster, Arts, Goldsmiths’, Northumbria) do. Is that fair?
Fairness
The question for anyone concerned with academic standards is whether these provider level differentials are fair. The OfS algorithm – as noted above – uses age, prior attainment, and subject as study as explicatory factors. It’s worth dealing with each in turn.
- OfS reckons that students with less than stellar A levels are less likely to get good degrees than those with AAA or above– so providers who recruit other kinds of learner will be penalised by the algorithm no matter how good they are at treating non-traditional learners.
- Age doesn’t quite work how you might expect – mature students are very slightly more likely to get a first or an upper second than the traditional 18 year old entry cohort.
- And humanities or social sciences subjects are judged to be harder to get a first in than physical sciences: so if you have (say) a huge law school and not many chemists you will struggle with the output of this algorithm.
I’d love to show you the standard errors and p-values that offer reassurance on the quality of this information here, but I understand from OfS that there was an issue with calculating them correctly: the figures have now been removed from the annex. The team are satisfied that the coefficients are accurate for what that’s worth, but if you end up being investigated as a result of this data I would be asking some questions here.
OfS has arrived at these insights through analysis of previous years of data – and this is a valid thing for them to have done. The failure to predict so much of what has actually happened suggests to me that other assumptions should be added to the model. It used to have disability, ethnicity, sex, and TUNDRA – these were axed from the model in 2023 ostensibly because they didn’t have much explanatory value.
There is a commendable clarity in the technical annex that any gap between the model and reality is because of “a result of unobserved effects between academic years that have not been accounted for and have not been included as explanatory variables in the model”. It is good to see language like that up top too, as a counterbalance to the rather accusatory term “unexplained”.
What of it?
We wrote about the three investigations that have thus far come about as a result of this data when we got the reports published last year. What was notable from those judgements was that OfS did not find any current evidence of grade inflation at any of the three providers involved, though at two of the three they did find a historic concern about degree algorithm that had been in place prior to the existence of the OfS and was addressed speedily when it became apparent that it was causing problems.
I am going to stick my neck out and say that there are likely to be no providers that are carrying out deliberate and systematic grade inflation as a matter of policy. If OfS feels that there are things providers are innocently doing that may result in grades being less than reliable what it needs to do is provide causal and statistical evidence that this is the case – and it will find this easier if it works with providers in the spirit of enhancement and continuous improvement rather than playing to the headlines.