Category: LEO

  • Is there a place for LEO in regulation?

    Is there a place for LEO in regulation?

    The OfS have, following a DfE study, recently announced a desire to use LEO for regulation. In my view this is a bad idea.

    Don’t get me wrong, the Longitudinal Outcomes from Education (LEO) dataset is a fantastic and under-utilised tool for historical research. Nothing can compare to LEO for its rigour, coverage and the richness of the personal data it contains.

    However, it has serious limitations, it captures earnings and not salary, for everyone who chooses to work part time it will seriously underestimate the salary they command.

    And fundamentally it’s just too lagged. You can add other concerns around those choosing not to work and those working abroad if you wish to undermine its utility further.

    The big idea

    The OfS is proposing using data from 3 years’ after graduation which I assume to mean the third full tax year after graduation although it could mean something different, no details are provided. Assuming that my interpretation is correct the most recent LEO data published in June this year relates to the 2022-23 tax year so for that to be the third full tax year after graduation (that’s the that’s the 2018-19 graduating cohort, and even if you go for the third tax year including the one they graduated in it’s the 2019-20 graduates). The OfS also proposes to continue to use 4 year aggregates which makes a lot of sense to avoid statistical noise and deal with small cohorts but it does mean that some of the data will relate to even earlier cohorts.

    The problem is therefore if the proposed regime had been in place this year the OfS would have just got its first look at outcomes from the 2018-19 graduating cohort who were of course entrants in 2016-17 or earlier. When we look at it through this lens it is hard to see how one applies any serious regulatory tools to a provider failing on this metric but performing well on others especially if they are performing well on those based on the still lagged but more timely Graduate Outcomes survey.

    It is hard to conceive of any courses that will not have had at least one significant change in the 9 (up to 12!) years since the measured cohort entered. It therefore won’t be hard for most providers to argue that the changes they have made since those cohorts entered will have had positive impacts on outcomes and the regulator will have to give some weight to those arguments especially if they are supported by changes in the existing progression, or the proposed new skills utilisation indicator.

    A problem?

    And if the existing progression indicator is problematic then why didn’t the regulator act on it when it had it four years earlier? The OfS could try to argue that it’s a different indicator capturing a different aspect of success but this, at least to this commentators mind, is a pretty flimsy argument and is likely to fail because earnings is a very narrow definition of success. Indeed, by having two indicators the regulator may well find themselves in a situation where they can only take meaningful action if a provider is failing on both.

    OfS could begin to address the time lag by just looking at the first full tax year after graduation but this will undoubtedly be problematic as graduates take time to settle into careers (which is why GO is at 15 months) and of course the interim study issues will be far more significant for this cohort. It would also still be less timely than the Graduate Outcomes survey which itself collects the far more meaningful salary rather than earnings.

    There is of course a further issue with LEO in that it will forever be a black box for the providers being regulated using it. It will not be possible to share the sort of rich data with providers that is shared for other metrics meaning that providers will not be able to undertake any serious analysis into the causes of any concerns the OfS may raise. For example, a provider would struggle to attribute poor outcomes to a course they discontinued, perhaps because they felt it didn’t speak to the employment market. A cynic might even conclude that having a metric nobody can understand or challenge is quite nice for the OfS.

    The use of LEO in regulation is likely to generate a lot of work for the OfS and may trigger lots of debate but I doubt it will ever lead to serious negative consequences as the contextual factors and the fact that the cohorts being considered are ancient history will dull, if not completely blunt, the regulatory tools.

    Richard Puttock writes in a personal capacity.

    Source link

  • LEO has a role to play in picking winners

    LEO has a role to play in picking winners

    Perhaps we have LEO all wrong?

    In the week of the industrial strategy, which brought confirmation as to how central higher level skills will be to our plans for national growth, the annual release of longitudinal education outcomes (LEO) data) hits a bit different.

    In the past, ministers and higher education obsessives, have seen the release as a chance to rank universities and subjects – and perhaps even a chance to spot the infamous “low quality courses” – based on median graduate earnings. As we’ve been through time and time again on Wonkhe, this doesn’t really work.

    But quietly, and at the instigation of the former Unit for Future Skills which now forms a part of Skills England, earnings have taken something of a back seat to more detailed information about subject areas, place, and industrial sectors. If you wanted to understand the way in which investment in particular subjects might translate to benefits to particular industries (to support changes to the allocation of the Office for Students Strategic Priorities Grant, for example), LEO is where you would look.

    Factory model

    And you can illustrate this with today’s release of LEO data on graduate outcomes and provider level data. (There’s even a new DfE dashboard.)

    Let’s take a worked example. Advanced manufacturing is one of the eight sectors named in the industrial strategy as a national priority. As a sector it is going to be receiving £4.3bn over the next five years, including £2.8bn in research and development, from the state. Of the “frontier industries” within the manufacturing sector battery technology and advanced materials feel like clear priorities – and exactly the kind of things that might need the higher level skills that a university education might bring – although manufacturing as a whole will be in a similar position.

    How do you develop these skill sets?

    First up, we can see what graduates that work in manufacturing industries have studied. Engineering is the main route (and the route to the kind of high earnings that suggest senior roles) – just over 1,900 people with a first degree in engineering are working in the sector.

    [Full screen]

    But it is clear that manufacturing also needs business expertise (there are 1,350 business graduates in the field) and design capability (890 creative arts and design graduates). Both these groups earn rather less than the engineers – the low level of pay for creative arts graduates suggests that these workers may have recently moved into the sector or are in non-graduate roles, or (whisper it) we might need to pay them better. This latter interpretation is backed up by the fact that similar proportions of graduates are in this sector a year after graduation, and designers are consistently paid less.

    There is a regional component to this too (using the purple filters at the bottom of the chart, note that you can only use one of these at once) as we can see that when we look at London we can see that salary differentials shrink between our top three.

    In terms of specific manufacturing specialisms giving design graduates good jobs ten years on, another plot of the same data suggests automotive, and aircraft/space craft, manufacturing are particularly positive destinations. That said, they are not the primary destinations for creative arts and design graduates ten years out: as with many subject areas teaching is well represented, as are retail and advertising.

    [Full screen]

    The regional dimension

    Let’s say that we also want to use state investment in skills for manufacturing to have a regional benefit – and we wanted to focus on manufacturing in the north east. Would we need to specifically grow design provision in the north east of England in order to provide design graduates for firms there?

    Based on the data, the answer is yes. Most design graduates who studied in the north east working in manufacturing, are working in the north east ten years on. Though the numbers are small, the same appears to be true at other career points.

    [Full screen]

    Do we know if design students (rather than creative arts students) are the ones working in manufacturing? We don’t for certain but we can be reasonably sure. There are more design students than any other subjects in the creative arts and design area – and they tend to earn more ten years on than their peers in the arts. The median salaries also seem to match up – designers aren’t big earners, but they do earn more than some more traditional “skills priority” areas like ecology and mineral technology.

    [Full screen]

    Picking winners

    Seasoned LEO-watchers will be aware that there are issues when you look at specific sub-groups within the data. One of the drawbacks here is that we can’t track design specifically by provider – it would be helpful, if we decide we need more product designers in the north east, we knew which existing provider was landing them that well paid senior roles later in their careers. Best we can tell (and only five years out) it might be Newcastle and Northumbria – but we also need to factor in more granular effects (is Newcastle more likely to offer higher-paid design jobs than Sunderland or Middlesborough – I’d expect so, but we don’t get that data).

    [Full screen]

    It’s also likely that Teesside and Sunderland will be recruiting students with less traditional backgrounds, and we know that will have an impact on careers and salaries – but interestingly it appears to be Northumbria that are getting the best results for students from a domicile in a POLAR4 quintile one area.

    So there we have it: if we want to invest in advanced manufacturing in the north east of England in a way that gets the best results for local people, we need to support the design courses at Northumbria University. Which is something of a shame as we cut state support for design provision a couple of years ago.

    Source link

  • Can we use LEO in regulation?

    Can we use LEO in regulation?

    The Institute for Fiscal Studies answers the last government’s question on earnings data in regulation. David Kernohan reads along

    Source link