Tag: dark

  • Ed data goes dark: Why it matters (opinion)

    Ed data goes dark: Why it matters (opinion)

    When President Donald Trump and Elon Musk’s Department of Government Efficiency set out to slash billions from the federal budget, it puzzled me as to why one of their first targets was an obscure data collection and research agency, the Institute of Education Sciences, a relatively modest operation buried deeply in the corridors of the Department of Education, and indeed one few had ever heard of. Since then, the newly installed secretary of education has ordered a review of all the department’s functions as part of what she ominously called the department’s “momentous final mission.”

    A conversation with a trusted colleague helped me understand the cuts to IES, noting that the action should be seen as part of a new breed of autocrats around the world who seek to control information to hide the impacts of their actions from the public. In contemporary authoritarian governments, control of information—or what has come to be known today as informational autocracy—often substitutes for brute force.

    Similar to how the Trump administration is seizing control of the White House press pool, canceling contracts for independent, high-quality education research is another way of controlling information. As Democratic lawmakers wrote in a Feb. 21 letter decrying the cuts, “The consequences of these actions will prevent the public from accessing accurate information about student demographics and academic achievement, abruptly end evaluations of federal programs that ensure taxpayer funds are spent wisely, and set back efforts to implement evidence-based reforms to improve student outcomes.”

    IES houses a vast warehouse of the nation’s education statistics. Data collected by the agency is used by policymakers, researchers, teachers and colleges to understand student achievement, enrollment and much more about the state of American education. With IES being among the largest funders of education research, cutting it limits public access to what’s happening in the nation’s schools and colleges.

    Claiming to eliminate waste and corruption, Musk’s first round of cuts involved canceling what DOGE initially said were nearly $900 million in IES contracts (though, as subsequent reporting has since revealed, DOGE’s math doesn’t add up and the canceled contracts seem to amount to much less). A second round purportedly sliced another $350 million in contracts and grants. It’s unclear how much more is destined to be chopped, since these may only be the first in a series of cuts designed to completely dismantle the Education Department. Though a department spokesperson initially said that the cuts would not affect the National Assessment of Educational Progress, a standardized test known as the nation’s report card, and the College Scorecard, which allows citizens to search for and compare information about colleges, we’ve since seen the cancellation of a national NAEP test for 17-year-olds.

    In the Obama years, public data helped reveal bad actors among for-profit colleges, which were receiving millions in federal aid while delivering inferior education to poor and working-class students who yearned for college degrees. Since so few actually completed, what many got instead was crushing college debt. Luckily, good data helped drive nearly half of all for-profit programs to shut down. Publicly disseminated data exposes where things go wrong. But you can’t track down con men without evidence.

    Ideally, in a well-functioning democracy, with a richly informed public, data helps us reach informed decisions, leading to greater accountability and enabling us to hold officials responsible for their actions. With access to reliable information about what’s happening behind closed doors, data helps us understand what may be going on, even to protest actions we may oppose.

    Lately, however, things aren’t looking good. Since Trump and his top officials have slashed race-conscious programs and moved to prohibit funding for certain areas of research, higher ed leadership has remained mostly silent, with only a handful of college presidents protesting. Most have shrunk into the wings, cowed by Trump’s power to defund institutions. It already has the eerie feeling of watching your step.

    Shutting down potentially revealing data collection is perhaps the least worrisome page in an autocrat’s playbook. As Trump continues to follow the authoritarian path set by leaders in Hungary, Turkey and elsewhere, we should expect other, more damaging and more frightening higher ed moves that have been imposed by other autocrats—selecting college presidents, controlling faculty hiring and advancement, punishing academic dissent, imposing travel restrictions.

    Just a few months ago, there was comfort in knowing everything was there—data on enrollments, graduation rates, participation rates of women and other groups. All very neatly organized and accessible whenever you wanted. Even though some found IES technology old and clunky, it felt like higher ed was running according to a reliable scheme, that you could go online and open data files as in a railroad timetable. Without it, there might be a train wreck ahead and you wouldn’t know it until it was too late. Now these luxurious numbers may soon be lost, with decades of America’s academic history pitched into digital darkness.

    It’s frightening to realize that we’ll no longer be operating on solid intelligence. That we’ll no longer have guideposts, supported by racks of sensibly collected numbers to tell us if we’re on the right path or if we’re far afield. Trump’s wrecking ball has smashed our confidence, a confidence built on years of reliable data. We’ll soon be in the dark.

    Robert Ubell is vice dean emeritus of online learning at New York University’s Tandon School of Engineering and senior editor of CHLOE 9, the ninth national survey of higher ed chief online learning officers. A collection of his essays on virtual education, Staying Online: How to Navigate Digital Higher Education, was published by Routledge.

    Source link

  • The data dark ages | Wonkhe

    The data dark ages | Wonkhe

    Is there something going wrong with large surveys?

    We asked a bunch of people but they didn’t answer. That’s been the story of the Labour Force Survey (LFS) and the Annual Population Survey (APS) – two venerable fixtures in the Office for National Statistics (ONS) arsenal of data collections.

    Both have just lost their accreditation as official statistics. A statement from the Office for Statistical Regulation highlights just how much of the data we use to understand the world around us is at risk as a result: statistics about employment are affected by the LFS concerns, whereas APS covers everything from regional labour markets, to household income, to basic stuff about the population of the UK by nationality. These are huge, fundamental, sources of information on the way people work and live.

    The LFS response rate has historically been around 50 per cent, but it had fallen to 40 per cent by 2020 and is now below 20 per cent. The APS is an additional sample using the LFS approach – current advice suggests that response rates have deteriorated to the extent that it is no longer safe to use APS data at local authority level (the resolution it was designed to be used at).

    What’s going on?

    With so much of our understanding of social policy issues coming through survey data, problems like these feel almost existential in scope. Online survey tools have made it easier to design and conduct surveys – and often design in the kind of good survey development practices that used to be the domain of specialists. Theoretically, it should be easier to run good quality surveys than ever before – certainly we see more of them (we even run them ourselves).

    Is it simply a matter of survey fatigue? Or are people less likely to (less willing to?) give information to researchers for reasons of trust?

    In our world of higher education, we have recently seen the Graduate Outcomes response rate drop below 50 per cent for the first time, casting doubt as to its suitability as a regulatory measure. The survey still has accredited official statistics status, and there has been important work done on understanding the impact of non-response bias – but it is a concerning trend. The national student survey (NSS) is an outlier here – it has a 72 per cent response rate last time round (so you can be fairly confident in validity right down to course level), but it does enjoy an unusually good level of survey population awareness even despite the removal of a requirement for providers to promote the survey to students. And of course, many of the more egregious issues with HESA Student have been founded on student characteristics – the kind of thing gathered during enrollment or entry surveys.

    A survey of the literature

    There is a literature on survey response rates in published research. A meta-analysis by Wu et al (Computers in Human Behavior, 2022) found that, at this point, the average online survey result was 44.1 per cent – finding benefits for using (as NSS does) a clearly defined and refined population, pre-contacting participants, and using reminders. A smaller study by Diaker et al (Journal of Survey Statistics and Methodology, 2020) found that, in general, online surveys yield lower response rates (on average, 12 percentage point lower) than other approaches.

    Interestingly, Holton et al (Human Relations, 2022) show an increase in response rates over time in a sample of 1014 journals, and do not find a statistically significant difference linked to survey modes.

    ONS itself works with the ESRC-funded Survey Futures project, which:

    aims to deliver a step change in survey research to ensure that it will remain possible in the UK to carry out high quality social surveys of the kinds required by the public and academic sectors to monitor and understand society, and to provide an evidence base for policy

    It feels like timely stuff. Nine strands of work in the first phase included work on mode effects, and on addressing non-response.

    Fixing surveys

    ONS have been taking steps to repair LFS – implementing some of the recontacting/reminder approaches that have been successfully implemented and documented in the academic literature. There’s a renewed focus on households that include young people, and a return to the larger sample sizes we saw during the pandemic (when the whole survey had to be conducted remotely). Reweighting has led to a bunch of tweaks to the way samples are chosen, and non-responses accounted for.

    Longer term, the Transformed Labour Force Survey (TLFS) is already being trialed, though the initial March 2024 plans for full introduction has been revised to allow for further testing – important given a bias towards older age group responses, and an increased level of partial responses. Yes, there’s a lessons learned review. The old LFS and the new, online first, TLFS will be running together at least until early 2025 – with a knock on impact on APS.

    But it is worth bearing in mind that, even given the changes made to drive up responses, trial TLFS response rates have been hovering around just below 40 per cent. This is a return to 2020 levels, addressing some of the recent damage, but a long way from the historic norm.

    Survey fatigue

    More usually the term “survey fatigue” is used to describe the impact of additional questions on completion rate – respondents tire during long surveys (as Jeong et al observe in the Journal of Development Economics) and deliberately choose not to answer questions to hasten the end of the survey.

    But it is possible to consider the idea of a civilisational survey fatigue. Arguably, large parts of the online economy are propped up on the collection and reuse of personal data, which can then be used to target advertisements and reminders. Increasingly, you now have to pay to opt out of targeted ads on websites – assuming you can view the website at all without paying. After a period of abeyance, concerns around data privacy are beginning to reemerge. Forms of social media that rely on a constant drive to share personal information are unexpectedly beginning to struggle – for younger generations participatory social media is more likely to be a group chat or discord server, while formerly participatory services like YouTube and TikTok have become platforms for media consumption.

    In the world of public opinion research the struggle with response rates has partially been met via a switch from randomised phone or in-person to the use of pre-vetted online panels. This (as with the rise of focus groups) has generated a new cadre of “professional respondents” – with huge implications for the validity of polling even when weighting is applied.

    Governments and industry are moving towards administrative data – the most recognisable example in higher education being the LEO dataset of graduate salaries. But this brings problems in itself – LEO lets us know how much income graduates pay tax on from their main job, but deals poorly with the portfolio careers that are the expectation of many graduates. LEO never cut it as a policymaking tool precisely because of how broadbrush it is.

    In a world where everything is data driven, what happens when the quality of data drops? If we were ever making good, data-driven decisions, a problem with the raw material suggests a problem with the end product. There are methodological and statistical workarounds, but the trend appears to be shifting away from people being happy to give out personal information without compensation. User interaction data – the traces we create as we interact with everything from ecommerce to online learning – are for now unaffected, but are necessarily limited in scope and explanatory value.

    We’ve lived through a generation where data seemed unlimited. What tools do we need to survive a data dark age?

    Source link