Tag: Program

  • DOGE fails to accurately disclose contract and program cuts

    DOGE fails to accurately disclose contract and program cuts

    As part of his administration’s broad push for government transparency, on Feb. 18 President Donald Trump ordered all federal agencies to publicize “to the maximum extent permitted by law” the complete details of every program, contract or grant they terminated.

    “The American people have seen their tax dollars used to fund the passion projects of unelected bureaucrats rather than to advance the national interest,” Trump wrote in the memo, tilted “Radical Transparency About Wasteful Spending.” “[They] have a right to see how the Federal Government has wasted their hard-earned wages.”

    Immediately after receiving a copy of the order, Inside Higher Ed reached out to the Department of Education and requested a comprehensive list of any and all such cuts, as well as explanations for why each contract was terminated. But two weeks later, the Education Department has yet to respond, and neither the department nor the staff it has partnered with from Elon Musk’s Department of Government Efficiency have publicly released any more information about the terminated contracts and grants.

    In fact, DOGE—the agency leading the crusade of cuts—has continuously made edits to the “Wall of Receipts,” where it was supposedly outlining the cuts that have been made. Late Sunday night, the group deleted hundreds of contracts it had previously claimed to cancel, The New York Times first reported and Inside Higher Ed confirmed.

    “It’s absolutely hypocrisy,” said Antoinette Flores, director of higher education accountability and quality at New America, a left-leaning think tank. “It feels like we’re all being gaslit. I don’t know why they are saying they want to be transparent without being transparent.”

    For weeks, higher education leaders, policy experts and advocates have raised concerns as the department terminated more than 100 assorted grants and research contracts. Combined, the cuts are purportedly valued at nearly $1.9 billion, according to the department, and will affect a swath of institutions, including the department’s largest research arm as well as regional labs and external nonprofits that collaborated with local officials to improve learner outcomes. Combined, the cuts will dramatically impact the data available to researchers and policymakers focused on improving teaching and learning strategies, experts say.

    Education scholars are worried that the cuts will leave state officials and college administrators with little evidence on which to base their strategies for student success and academic return on investment. One professor went as far as to say that the cuts are “an assault on the U.S.’s education data infrastructure.”

    And though the Trump administration has flaunted its transparency and glorified DOGE’s website as a prime example of their success in providing public records, policy experts on both sides of the political aisle say the collective contract value displayed is an overestimate. Calculating savings is more nuanced than just listing a contract’s maximum potential value, they say—and even if they saved money, some of the terminated programs were congressionally mandated.

    Over all, the sudden nature of the cuts, combined with the questionable accuracy of reported savings and a lack of further transparency, have left higher education advocacy groups deeply concerned.

    “The cuts that happened recently are going to have far-reaching impacts, and those impacts could really be long term unless some rapid action is taken,” said Mamie Voight, president of the Institute for Higher Education Policy, a national nonprofit that campaigns for college access and student success. “This information is useful and essential to help policymakers steward taxpayer dollars responsibly.”

    “To eliminate data, evidence and research is working in opposition to efficiency,” she later added.

    The department did not respond to requests for comment on Voight’s and Flores’s criticisms.

    A Data ‘Mismatch’

    For many in higher ed, the executive actions Trump has taken since January raise questions about executive overreach and government transparency. But Nat Malkus, deputy director of education policy studies at the American Enterprise Institute, a right-leaning think tank, said, “It’s not a matter of deception” or even simply a question of transparency.

    Instead, he said, “The question is, what’s the quality of the transparency? And what can we make of it?”

    In a recent analysis, titled “Running Down DOGE’s Department of Education Receipts,” Malkus compared a leaked list of the 89 terminated Institute of Education Sciences contracts, along with detailed data from USASpending.gov, to those DOGE had posted on its website. He said he found major inconsistencies, or a “mismatch” in how they defined the purported contract value.

    He also noted that though the “Wall of Receipts” has two separate tabs, one listing a contract’s value and another listing its savings, it displays the overall contract value first. The agency also declines to explain the difference between value and savings or how it calculates either.

    As is the case with contract values, DOGE has been inconsistent in how it calculates savings. But what the agency most often displays to the public is how much a contract could theoretically cost if all options and add-ons are utilized—known as the potential total—minus the amount the government had currently agreed to spend by the end of the contract, or the total obligation. So in other words, Malkus said, DOGE is sharing how much the government could save if it were to continue the contract and receive the promised deliverables without adding any extra bells and whistles.

    But that’s not what DOGE has done. Instead, it has terminated the contracts, and the Education Department won’t receive the final product it was paying for.

    To best represent savings in that scenario, Malkus said, DOGE would calculate the difference between how much the government had agreed to spend by the end of the contract—the total obligation—and how much the government has already spent, or the total outlay.

    “It’s weird because DOGE is publishing one set of savings that I don’t think actually makes sense to anybody, and they’re ignoring savings that they actually are conceivably getting,” Malkus said. “There are some good reasons that they might choose to do that. But DOGE would do well to explain what these dollars are, because right now, no one can tell.”

    Malkus first spoke with Inside Higher Ed on Friday. But since then, the DOGE database has changed. Malkus said Tuesday that some of the initial trends in the way DOGE appeared to be calculating savings are no longer present and he has yet to find a new, even semiconsistent formula for how DOGE is calculating savings.

    “The pace of change on DOGE’s numbers is dizzying even for someone like me who works at analyzing these receipts,” Malkus said. “Each week there have been changes to the number of contracts and within contracts the values and savings that DOGE is publishing. It’s hard to know if they are trying to get this right, because it’s impossible to find a consistent trail.”

    I don’t attribute it to a desire to falsely advertise transparency and not deliver on it. I just think they need to do a much better job in the execution.”

    —Nat Malkus

    And even if there were a consistent, uniform formula for how DOGE officials are calculating efficiency, in some cases they still choose to highlight overall contract value rather than the direct savings. For example, a DOGE social media post about the Institute of Education Sciences cuts noted the contracts were worth $881 million in total.

    “So are the actual savings equal to that implied? No, they are not,” Malkus said. “They are far, far less than that amount, somewhere around 25 percent of the total.”

    The agency’s website doesn’t detail the team’s methodology or offer any explanation about why the cuts were made. Malkus believes this lack of clarification reflects the Trump administration’s effort to make notable cuts quickly. He added that while he doesn’t agree with every cut made, he understands and supports the “aggressiveness” of Trump and Musk’s approach.

    “If they don’t move quickly, then there’s commissions, and then you have to go to the secretary, and they have interminable meetings and everything gets slowed down,” he said. “So one of their priorities is to move fast, and they don’t mind breaking things in the process.”

    From Malkus’s perspective, the inconsistencies in how each cut is documented, the many edits that have been made to the DOGE database and the lack of explanation for each cut isn’t a matter of “malice or dishonesty,” but rather “mistakes.”

    “I don’t think their savings are a clear estimation of what taxpayers are actually saving. But I don’t attribute it to a desire to falsely advertise transparency and not deliver on it. I just think they need to do a much better job in the execution,” he said.

    A ‘Disregard for the Law’

    Flores from New America conducted similar research and, like Malkus, found that the DOGE data doesn’t add up and exaggerates the savings. However, she had different views on the cause and effects of the agency’s aggressive, mistake-riddled approach.

    “It’s like taking a wrecking ball to important government services,” she said. “If you’re trying to be efficient, you should take into consideration how far along is a contract? How much have we spent on this? Are we getting anything for the investment we’ve made?”

    The Trump administration has broadly explained its cuts as a response to the “liberal ideology” of diversity, equity and inclusion and an effort to increase efficiency. But to Flores, they target anything but “waste, fraud and abuse.”

    “The reason why the Trump administration says it wants to eliminate the Department of Education is because you don’t see improvement in student performance,” she said. “But if you want to improve student performance, you need to understand what is happening on the ground with students and evidence-based research on how to help students improve.”

    And many of the studies affected by the contract cuts were nearly completed, she said. They were projects on which the agency had already spent hundreds of millions of dollars. So by cutting them now, the department loses the data and wastes taxpayer funds.

    It’s absolutely hypocrisy. It feels like we’re all being gaslit.”

    —Antoinette Flores

    “I’ve talked to some researchers who worked at one of the organizations that had their contracts cut, and they said all work has to stop,” she said. “No matter how close it was to being finished, it just has to stop.”

    Flores also noted that some of the studies terminated via contract cuts—particularly the National Postsecondary Student Aid Study—are congressionally mandated, so ending them is unconstitutional.

    “The people making these cuts don’t necessarily understand the math. They don’t necessarily understand the contracts or the purpose of them, and there’s a disregard for the law,” she said.

    Voight from IHEP agreed, describing projects like NPSAS as “core data sets that the field relies upon.”

    “Lawmakers often turn to these types of longitudinal and sample studies to answer questions that they have as they’re trying to build policies. And states turn to this type of information to help them benchmark how they’re faring against national numbers,” she said. “So these studies themselves are a really, really devastating loss.”

    Even some contracts that weren’t cut will suffer consequences, Voight noted. For example, though the Statewide Longitudinal Data Systems grant program has so far been shielded from outright termination, she said, it didn’t come away entirely unscathed. The data systems rely on key information from a program called Common Education Data Standards, which was slashed; without CEDS, the grant program won’t be nearly as effective.

    “The cuts have actually been misunderstanding the interrelationships between many of these different products,” Voight said.

    Over all, she believes the Department of Education, and specifically IES, are not the best places for efficiency cuts. The $807.6 million budget for the Institute of Education Sciences in fiscal year 2024 is just “a drop in the bucket” compared with the amount spent on other research and development groups, like the $4.1 billion given to the Defense Advanced Research Projects Agency the same year.

    “To think about how to build efficiencies is certainly not a bad question to ask. But IES is already such a lean operation, and the way that they are trying to build evidence is critical,” she said. “So we should really be focusing on investment in our education research infrastructure and taking a strategic approach to any changes that are going to be made.”

    Source link

  • Indiana First Lady to Raise Money for Dolly Parton’s Library Program – The 74

    Indiana First Lady to Raise Money for Dolly Parton’s Library Program – The 74


    Get stories like this delivered straight to your inbox. Sign up for The 74 Newsletter

    After slashing a popular reading program from the budget, Gov. Mike Braun said Friday he asked First Lady Maureen Braun to spearhead an initiative to keep Dolly Parton’s Imagination Library in Indiana.

    “She has agreed and she will work with philanthropic partners and in consultation with state leadership to identify funding opportunities for the book distribution program,” the governor said in a news release.

    The program gifts free, high quality, age-appropriate books to children from birth to age five on a monthly basis, regardless of family income.

    Former Gov. Eric Holcomb included a statewide expansion of the program in his 2023 legislative agenda. The General Assembly earmarked $6 million for the program in the state’s last biennial budget — $2 million in the first year and $4 million in the second — to ensure that all Hoosier kids qualify to receive free books.

    But when Gov. Braun prepared his budget proposal in January he discontinued the funding as part of an overall effort to rein in state spending.

    “I am honored to lead this work to help ensure our youngest Hoosiers have as much exposure as possible to books and learning,” said First Lady Maureen Braun. “Indiana has many strong community partners and I am confident we will collaborate on a solution that grows children’s love of reading.”

    Jeff Conyers, president of The Dollywood Foundation, said he appreciates Braun’s commitment to early childhood literacy.

    “The Imagination Library brings the joy of reading to over 125,000 Hoosier children each month in all 92 counties across the state, and we are encouraged by Governor and First Lady Braun’s support to ensure its future in Indiana. We look forward to working with the Governor and First Lady, state leaders, and Local Program Partners to keep books in the hands of Indiana’s youngest learners and strengthen this foundation for a lifetime of success,” he said.

    Indiana Capital Chronicle is part of States Newsroom, a nonprofit news network supported by grants and a coalition of donors as a 501c(3) public charity. Indiana Capital Chronicle maintains editorial independence. Contact Editor Niki Kelly for questions: [email protected].


    Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter

    Source link

  • N.C. community colleges launch program modeled on CUNY ASAP

    N.C. community colleges launch program modeled on CUNY ASAP

    The North Carolina Community College System is launching NC Community Colleges Boost, a new program to move students into high-demand careers in the state. The program is modeled after the City University of New York’s Accelerated Study in Associate Programs, or CUNY ASAP, known for offering extensive wraparound supports for low-income students to increase their completion rates, including personalized academic advising and covering various college costs.

    The program will launch at eight community colleges across the state in 2025 and at seven more colleges the following year, with the help of the CUNY ASAP National Replication Collaborative, which has helped other institutions create their own versions of the heavily studied and rapidly spreading program. Participating North Carolina students will have to be in fields of study that lead to high-demand careers in the state, among other eligibility criteria.

    The CUNY ASAP model is “the gold standard for increasing completion in higher education,” North Carolina Community College System president Jeff Cox said in an announcement Wednesday. “In the NC Community Colleges Boost implementation, we have taken that model and aligned it with North Carolina’s workforce development goals as specified in the PropelNC initiative,” the system’s new funding model intended to better align funding with workforce needs.

    The effort is supported by a grant of about $35.6 million from the philanthropy Arnold Ventures, the largest private grant ever received by the North Carolina Community College System.

    “This program has increased graduation rates, reduced time to graduation, and lowered the cost per graduate across many individual colleges in several states,” Cox said of CUNY ASAP. “Here in North Carolina, we have every reason to expect similar results.”

    Source link

  • Harvard lays off staff at its Slavery Remembrance Program

    Harvard lays off staff at its Slavery Remembrance Program

    Harvard University last week laid off the staff of the Harvard Slavery Remembrance Program, who were tasked with identifying the direct descendants of those enslaved by Harvard-affiliated administrators, faculty and staff, The Boston Globe reported.

    The work, which was part of the university’s $100 million Legacy of Slavery initiative, will now fall entirely to American Ancestors, a national genealogical nonprofit that Harvard was already partnering with, according to a news release.

    A Harvard spokesperson declined to comment on the layoffs to the Globe.

    The Harvard Crimson first reported the news, noting that the HSRP staff were terminated without warning Jan. 23.

    Protesting the move, Harvard history professor Vincent Brown resigned from the Legacy of Slavery Memorial Project Committee, which was assigned the task of designing a memorial to those enslaved by members of the Harvard community.

    Brown wrote in his resignation letter, which he shared with Inside Higher Ed, that he had recently returned from a productive research trip to Antigua and Barbuda when he “learned that the entire [HSRP] team had been laid off in sudden telephone calls with an officer in Harvard’s human resources department.” He called the terminations “vindictive as well as wasteful.”

    “I hope and expect that the H&LS initiative will weather this latest controversy,” Brown wrote. “I only regret that I cannot formally be a part of that effort.”

    Harvard’s Legacy of Slavery Initiative has repeatedly come under fire since it was announced in 2022. Critics assailed its lack of progress last year. The two professors who co-chaired the memorial committee resigned last May, citing frustration with administrators; the executive director of the initiative, Roeshana Moore-Evans, followed them out the door. Then HSRP founding director Richard Cellini told the Crimson last fall that vice provost Sara Bleich had instructed him “‘not to find too many descendants.’”

    A university spokesperson denied that charge, telling the Crimson, “There is no directive to limit the number of direct descendants to be identified through this work.”

    Cellini was among those fired from the HSRP last week.

    Source link

  • Student Booted from PhD Program Over AI Use (Derek Newton/The Cheat Sheet)

    Student Booted from PhD Program Over AI Use (Derek Newton/The Cheat Sheet)


    This one is going to take a hot minute to dissect. Minnesota Public Radio (MPR) has the story.

    The plot contours are easy. A PhD student at the University of Minnesota was accused of using AI on a required pre-dissertation exam and removed from the program. He denies that allegation and has sued the school — and one of his professors — for due process violations and defamation respectively.

    Starting the case.

    The coverage reports that:

    all four faculty graders of his exam expressed “significant concerns” that it was not written in his voice. They noted answers that seemed irrelevant or involved subjects not covered in coursework. Two instructors then generated their own responses in ChatGPT to compare against his and submitted those as evidence against Yang. At the resulting disciplinary hearing, Yang says those professors also shared results from AI detection software. 

    Personally, when I see that four members of the faculty unanimously agreed on the authenticity of his work, I am out. I trust teachers.

    I know what a serious thing it is to accuse someone of cheating; I know teachers do not take such things lightly. When four go on the record to say so, I’m convinced. Barring some personal grievance or prejudice, which could happen, hard for me to believe that all four subject-matter experts were just wrong here. Also, if there was bias or petty politics at play, it probably would have shown up before the student’s third year, not just before starting his dissertation.

    Moreover, at least as far as the coverage is concerned, the student does not allege bias or program politics. His complaint is based on due process and inaccuracy of the underlying accusation.

    Let me also say quickly that asking ChatGPT for answers you plan to compare to suspicious work may be interesting, but it’s far from convincing — in my opinion. ChatGPT makes stuff up. I’m not saying that answer comparison is a waste, I just would not build a case on it. Here, the university didn’t. It may have added to the case, but it was not the case. Adding also that the similarities between the faculty-created answers and the student’s — both are included in the article — are more compelling than I expected.

    Then you add detection software, which the article later shares showed high likelihood of AI text, and the case is pretty tight. Four professors, similar answers, AI detection flags — feels like a heavy case.

    Denied it.

    The article continues that Yang, the student:

    denies using AI for this exam and says the professors have a flawed approach to determining whether AI was used. He said methods used to detect AI are known to be unreliable and biased, particularly against people whose first language isn’t English. Yang grew up speaking Southern Min, a Chinese dialect. 

    Although it’s not specified, it is likely that Yang is referring to the research from Stanford that has been — or at least ought to be — entirely discredited (see Issue 216 and Issue 251). For the love of research integrity, the paper has invented citations — sources that go to papers or news coverage that are not at all related to what the paper says they are.

    Does anyone actually read those things?

    Back to Minnesota, Yang says that as a result of the findings against him and being removed from the program, he lost his American study visa. Yang called it “a death penalty.”

    With friends like these.

    Also interesting is that, according to the coverage:

    His academic advisor Bryan Dowd spoke in Yang’s defense at the November hearing, telling panelists that expulsion, effectively a deportation, was “an odd punishment for something that is as difficult to establish as a correspondence between ChatGPT and a student’s answer.” 

    That would be a fair point except that the next paragraph is:

    Dowd is a professor in health policy and management with over 40 years of teaching at the U of M. He told MPR News he lets students in his courses use generative AI because, in his opinion, it’s impossible to prevent or detect AI use. Dowd himself has never used ChatGPT, but he relies on Microsoft Word’s auto-correction and search engines like Google Scholar and finds those comparable. 

    That’s ridiculous. I’m sorry, it is. The dude who lets students use AI because he thinks AI is “impossible to prevent or detect,” the guy who has never used ChatGPT himself, and thinks that Google Scholar and auto-complete are “comparable” to AI — that’s the person speaking up for the guy who says he did not use AI. Wow.

    That guy says:

    “I think he’s quite an excellent student. He’s certainly, I think, one of the best-read students I’ve ever encountered”

    Time out. Is it not at least possible that professor Dowd thinks student Yang is an excellent student because Yang was using AI all along, and our professor doesn’t care to ascertain the difference? Also, mind you, as far as we can learn from this news story, Dowd does not even say Yang is innocent. He says the punishment is “odd,” that the case is hard to establish, and that Yang was a good student who did not need to use AI. Although, again, I’m not sure how good professor Dowd would know.

    As further evidence of Yang’s scholastic ability, Dowd also points out that Yang has a paper under consideration at a top academic journal.

    You know what I am going to say.

    To me, that entire Dowd diversion is mostly funny.

    More evidence.

    Back on track, we get even more detail, such as that the exam in question was:

    an eight-hour preliminary exam that Yang took online. Instructions he shared show the exam was open-book, meaning test takers could use notes, papers and textbooks, but AI was explicitly prohibited. 

    Exam graders argued the AI use was obvious enough. Yang disagrees. 

    Weeks after the exam, associate professor Ezra Golberstein submitted a complaint to the U of M saying the four faculty reviewers agreed that Yang’s exam was not in his voice and recommending he be dismissed from the program. Yang had been in at least one class with all of them, so they compared his responses against two other writing samples. 

    So, the exam expressly banned AI. And we learn that, as part of the determination of the professors, they compared his exam answers with past writing.

    I say all the time, there is no substitute for knowing your students. If the initial four faculty who flagged Yang’s work had him in classes and compared suspicious work to past work, what more can we want? It does not get much better than that.

    Then there’s even more evidence:

    Yang also objects to professors using AI detection software to make their case at the November hearing.  

    He shared the U of M’s presentation showing findings from running his writing through GPTZero, which purports to determine the percentage of writing done by AI. The software was highly confident a human wrote Yang’s writing sample from two years ago. It was uncertain about his exam responses from August, assigning 89 percent probability of AI having generated his answer to one question and 19 percent probability for another. 

    “Imagine the AI detector can claim that their accuracy rate is 99%. What does it mean?” asked Yang, who argued that the error rate could unfairly tarnish a student who didn’t use AI to do the work.  

    First, GPTZero is junk. It’s reliably among the worst available detection systems. Even so, 89% is a high number. And most importantly, the case against Yang is not built on AI detection software alone, as no case should ever be. It’s confirmation, not conviction. Also, Yang, who the paper says already has one PhD, knows exactly what an accuracy rate of 99% means. Be serious.

    A pattern.

    Then we get this, buried in the news coverage:

    Yang suggests the U of M may have had an unjust motive to kick him out. When prompted, he shared documentation of at least three other instances of accusations raised by others against him that did not result in disciplinary action but that he thinks may have factored in his expulsion.  

    He does not include this concern in his lawsuits. These allegations are also not explicitly listed as factors in the complaint against him, nor letters explaining the decision to expel Yang or rejecting his appeal. But one incident was mentioned at his hearing: in October 2023, Yang had been suspected of using AI on a homework assignment for a graduate-level course. 

    In a written statement shared with panelists, associate professor Susan Mason said Yang had turned in an assignment where he wrote “re write it, make it more casual, like a foreign student write but no ai.”  She recorded the Zoom meeting where she said Yang denied using AI and told her he uses ChatGPT to check his English.

    She asked if he had a problem with people believing his writing was too formal and said he responded that he meant his answer was too long and he wanted ChatGPT to shorten it. “I did not find this explanation convincing,” she wrote. 

    I’m sorry — what now?

    Yang says he was accused of using AI in academic work in “at least three other instances.” For which he was, of course, not disciplined. In one of those cases, Yang literally turned in a paper with this:

    “re write it, make it more casual, like a foreign student write but no ai.” 

    He said he used ChatGPT to check his English and asked ChatGPT to shorten his writing. But he did not use AI. How does that work?

    For that one where he left in the prompts to ChatGPT:

    the Office of Community Standards sent Yang a letter warning that the case was dropped but it may be taken into consideration on any future violations. 

    Yang was warned, in writing.

    If you’re still here, we have four professors who agree that Yang’s exam likely used AI, in violation of exam rules. All four had Yang in classes previously and compared his exam work to past hand-written work. His exam answers had similarities with ChatGPT output. An AI detector said, in at least one place, his exam was 89% likely to be generated with AI. Yang was accused of using AI in academic work at least three other times, by a fifth professor, including one case in which it appears he may have left in his instructions to the AI bot.

    On the other hand, he did say he did not do it.

    Findings, review.

    Further:

    But the range of evidence was sufficient for the U of M. In the final ruling, the panel — comprised of several professors and graduate students from other departments — said they trusted the professors’ ability to identify AI-generated papers.

    Several professors and students agreed with the accusations. Yang appealed and the school upheld the decision. Yang was gone. The appeal officer wrote:

    “PhD research is, by definition, exploring new ideas and often involves development of new methods. There are many opportunities for an individual to falsify data and/or analysis of data. Consequently, the academy has no tolerance for academic dishonesty in PhD programs or among faculty. A finding of dishonesty not only casts doubt on the veracity of everything that the individual has done or will do in the future, it also causes the broader community to distrust the discipline as a whole.” 

    Slow clap.

    And slow clap for the University of Minnesota. The process is hard. Doing the review, examining the evidence, making an accusation — they are all hard. Sticking by it is hard too.

    Seriously, integrity is not a statement. It is action. Integrity is making the hard choice.

    MPR, spare me.

    Minnesota Public Radio is a credible news organization. Which makes it difficult to understand why they chose — as so many news outlets do — to not interview one single expert on academic integrity for a story about academic integrity. It’s downright baffling.

    Worse, MPR, for no specific reason whatsoever, decides to take prolonged shots at AI detection systems such as:

    Computer science researchers say detection software can have significant margins of error in finding instances of AI-generated text. OpenAI, the company behind ChatGPT, shut down its own detection tool last year citing a “low rate of accuracy.” Reports suggest AI detectors have misclassified work by non-native English writers, neurodivergent students and people who use tools like Grammarly or Microsoft Editor to improve their writing. 

    “As an educator, one has to also think about the anxiety that students might develop,” said Manjeet Rege, a University of St. Thomas professor who has studied machine learning for more than two decades. 

    We covered the OpenAI deception — and it was deception — in Issue 241, and in other issues. We covered the non-native English thing. And the neurodivergent thing. And the Grammarly thing. All of which MPR wraps up in the passive and deflecting “reports suggest.” No analysis. No skepticism.

    That’s just bad journalism.

    And, of course — anxiety. Rege, who please note has studied machine learning and not academic integrity, is predictable, but not credible here. He says, for example:

    it’s important to find the balance between academic integrity and embracing AI innovation. But rather than relying on AI detection software, he advocates for evaluating students by designing assignments hard for AI to complete — like personal reflections, project-based learnings, oral presentations — or integrating AI into the instructions. 

    Absolute joke.

    I am not sorry — if you use the word “balance” in conjunction with the word “integrity,” you should not be teaching. Especially if what you’re weighing against lying and fraud is the value of embracing innovation. And if you needed further evidence for his absurdity, we get the “personal reflections and project-based learnings” buffoonery (see Issue 323). But, again, the error here is MPR quoting a professor of machine learning about course design and integrity.

    MPR also quotes a student who says:

    she and many other students live in fear of AI detection software.  

    “AI and its lack of dependability for detection of itself could be the difference between a degree and going home,” she said. 

    Nope. Please, please tell me I don’t need to go through all the reasons that’s absurd. Find me one single of case in which an AI detector alone sent a student home. One.

    Two final bits.

    The MPR story shares:

    In the 2023-24 school year, the University of Minnesota found 188 students responsible of scholastic dishonesty because of AI use, reflecting about half of all confirmed cases of dishonesty on the Twin Cities campus. 

    Just noteworthy. Also, it is interesting that 188 were “responsible.” Considering how rare it is to be caught, and for formal processes to be initiated and upheld, 188 feels like a real number. Again, good for U of M.

    The MPR article wraps up that Yang:

    found his life in disarray. He said he would lose access to datasets essential for his dissertation and other projects he was working on with his U of M account, and was forced to leave research responsibilities to others at short notice. He fears how this will impact his academic career

    Stating the obvious, like the University of Minnesota, I could not bring myself to trust Yang’s data. And I do actually hope that being kicked out of a university for cheating would impact his academic career.

    And finally:

    “Probably I should think to do something, selling potatoes on the streets or something else,” he said. 

    Dude has a PhD in economics from Utah State University. Selling potatoes on the streets. Come on.

    Source link

  • 7 Trends to Inform Online Program Expansion in 2025

    7 Trends to Inform Online Program Expansion in 2025

    As I reviewed the new IPEDS data release last week, I was looking for the data and intelligence that would be most helpful for online enrollment leaders to have in hand to underpin and inform this year’s success. These points, in combination with key trends that became clear in other sources I reviewed late last year will enable online leaders to succeed this year as well as plan for the future.

    Note that I am not discussing changes that may emerge after January 20, but I will be doing so after a long talk I have scheduled with Cheryl Dowd from WCET who tracks online regulations and with whom I will be co-presenting at the RNL National Conference this summer.

    So, what do you need to know?

    1. Online and partially online enrollment continue to dominate growth.

    Four years after the pandemic, more students each year are continuing to decide to enroll in either fully or partially online study. While year-over-year change in every post-pandemic year has seen some “return to the classroom,” when compared with pre-pandemic enrollment (2019), 2.3 million more undergraduates and 450k more graduate students are choosing fully or partially online study. Perhaps more important, 3.2 million fewer undergraduates and 288k fewer graduate students are choosing classroom-only programs. Institutions seeking to grow enrollment must develop processes to quickly determine the best online programs to offer and get them “to market” within 12 months.

    Chart showing the pandemic transformed student preferences as millions of additional students chose online and partially online study

    2. Institutions seeking to grow online enrollment are now competing with non-profit institutions.

    As recently as five years ago, your strongest competition came from for-profit institutions. In some ways, these institutions were easy to beat (excepting their huge marketing budgets). They had taken a beating in the press to the extent that students knew about it, and they were far away and unknown. Today, institutions face no less of a competitive environment, but the institutions dominating the scene – and most likely a students’ search results – are national non-profits. These institutions are, of course, not local so they aren’t well known, but they have not been through the scrutiny which eroded interest in the for-profits. Student search engine results are also now filled with ambitious public and private institutions seeking to “diversity their revenue streams.” As such, institutional marketers need to adjust their strategies focused on successfully positioning their programs in a crowded market, knowing that they can “win” the student over the national online providers if they ensure that they rise to the top of search results.

    Graph showing national non-profits have taken the lead from for-profit institutions.Graph showing national non-profits have taken the lead from for-profit institutions.

    3. Online enrollment growth is being led by non-profit institutions.

    Seventeen of the 20 institutions reporting the greatest growth in online enrollment over the last five years are nonprofit institutions—a mix of ambitious public and private institutions and national non-profits. What is more, the total growth among institutions after the two behemoths far exceeds Southern New Hampshire University and Western Governors University. These nimble and dynamic institutions include a variety of institution types (with community colleges well represented) across the higher education sector. Institutions seeking to grow online enrollment should research what these institutions are offering and how they are positioning their programs in the market and emulate some of their best practices.

    Chart showing that the greatest online growth is among non-profit colleges.Chart showing that the greatest online growth is among non-profit colleges.

    4. New graduate program growth is dominated by online/partially online offerings.

    In 2024, a research study by Robert Kelchen documented growth in the number of available master’s programs in the U.S. over the last 15 years. Not only did Kelchen document a massive expansion in availability (over the 15-year period, institutions launched nearly 14,000 new master’s programs on a base of about 20,000), but also that the pace of launching online or hybrid programs dramatically outpaces classroom programs. This rise in available offerings far outpaces the rate of growth of the online student market, resulting in significantly higher levels of competition for each online student. Institutions seeking to grow their online footprint must ensure that they fully understand both the specific demand dynamics for each of their programs and the specifics of what online students want in their program. A mismatch on either factor will inhibit growth.

    Graph showing online/hybrid programs are driving new program development.Graph showing online/hybrid programs are driving new program development.

    5. Online success is breeding scrutiny of outcomes.

    We all know something of the power of social media today. This was reinforced for me recently by an Inside Higher Education story which focused on the 8-year rates of degree completion among the biggest online providers. The story was triggered by a widely read Linked IN post and followed up by numerous other stories and posts and comments across the platform. This is just the kind of exposure that is most likely to generate real scrutiny of the outcomes of online learning – which were already taking shape over the last year or more. In fact, this focus on outcomes ended up as one of the unfulfilled priorities of the Biden Education Department. I have long said that institutions seeking to enter the online space have an opportunity to tackle some of the quality issues that first plagued the for-profits, now challenge the national online non-profits, and will confront others if not addressed soon.

    Images showing online skeptics are raising concerns about completion rates among larger online providers.Images showing online skeptics are raising concerns about completion rates among larger online providers.

    6. Key preferences for online study have been changed by the pandemic.

    RNL’s own 2024 online student survey surfaced dozens of important findings that online leaders should consider as they chart their course. Two findings stand out as reflecting profound changes in online student preferences, and both are likely the result of pandemic-era experiences. First, all but 11 percent of online students told us that they are open to at least some synchronous activities in their program, likely the result of hundreds of online meetings during the pandemic. Similarly, they told us that the ideal time to communicate with recruiters/counselors from online programs is now during business hours. This is also likely to be related to the pandemic period, in which millions of people working from home began to regularly contend with some personal business during their day. Institutions should assess both of these factors as they think through student engagement (to address point #5), and the intense competition of the online space (to address point #3).

    Pie charts showing how pandemic experiences have shaped student preferences for synchronous/asynchronous classes and when to follow-upPie charts showing how pandemic experiences have shaped student preferences for synchronous/asynchronous classes and when to follow-up

    7. Contracting institutions are not focusing on online enrollment.

    Finally, we return to the new IPEDS data to see that institutions that have experienced the greatest enrollment contraction over the last five years demonstrate almost no access to fully online study (dark blue lines in the chart below), and only limited access to programs in which students can enroll in both online and classroom courses (light blue lines). Even where there has been some online or partially online growth, these efforts have not been given adequate attention to counterbalance contraction among students enrolled in classroom-only programs (green lines). These data again make it clear (as stated in point #1) that institutions facing classroom-only contraction must either amend their goals to account for reduced enrollment or determine which online or hybrid programs would be most attractive to students in their region and then ensure that such offerings are visible in a highly competitive higher education market.

    Chart showing contracting institutions are not focusing on online.Chart showing contracting institutions are not focusing on online.

    Explore more at our webinar

    Webinar: 5 Enrollment Trends You Need to Know for 2025Webinar: 5 Enrollment Trends You Need to Know for 2025

    Join us for a deeper dive into trends during our webinar, 5 Enrollment Trends You Need to Know for 2025. This discussion with me and a number of my RNL expert colleagues will look at research and trends that should shape strategic and tactical planning over the next 12 months. Particularly, as we enter what has been identified as the first year of the “demographic cliff,” data-informed decision-making will be more important to enrollment health than ever before. Register now.

    Source link

  • New Program Strategy: Go Deep, Not Wide

    New Program Strategy: Go Deep, Not Wide

    How to Strategically Expand Your Online Adult Degree Programs

    So you’ve built a successful online adult degree program. No small feat. Now you need to keep your foot on the gas to keep the momentum going. 

    Your first instinct might be to “go wide” with your program expansion strategy by launching a variety of new, unrelated programs to pair with your successful offering. While this diversification strategy might reap great rewards for consumer packaged goods giants like Unilever and Procter & Gamble, higher education is different. Your institution is different.  

    I find myself making the following recommendation over and over again when it comes to expanding online degree programs: Go deep, not wide. 

    This means building upon the success of your existing program by developing specialized offerings within the same field. The “go deep” method might not be the most popular, but in my experience, it’s often the most effective. Let’s break it down further — or should I say, dig deeper — to see if this approach is right for your school. 

    What Does Going Wide Mean for Your Online Adult Degree Programs?

    Let’s start with a hypothetical example: You have established a successful online Master of Business Administration (MBA) program with a positive reputation in the region. 

    Recently, you’ve heard cybersecurity and nursing degree programs are experiencing industry growth, so you decide to pursue programs in those areas next to build out a wider range of offerings. 

    Unfortunately, this strategic path can be a mistake. Here’s why: 

    However, expanding within the existing framework of business administration can allow for the amplification of this established brand equity, rather than starting from scratch with each new offering.

    Why Going Deep Is More Effective

    In higher education, the smart, strategic allocation of resources is crucial. You could put your institution’s limited resources toward a whole new program, such as a Bachelor of Science in Nursing (BSN) program or a Master of Science in Cybersecurity program. Or, you could just attach a new or adjacent offering to your successful online MBA program to channel your resources into an established program realm. 

    Forget efficacy for a moment. Which strategy sounds more efficient? 

    The good news is that going deep in one area of program offerings is often more effective and efficient. Instead of developing an entirely new adult degree program from scratch, you can simply add value to your existing online business program. 

    This might come in the form of added concentration options, such as MBA concentrations in entrepreneurship, accounting, finance, marketing, management, or strategic communications. 

    It could also involve adding another relevant degree program within the same area of study. For example, since you’re seeing a lot of success with your MBA program, you could add a finance or accounting degree program to build on the success and reputation of the established program.

    Key Benefits of Going Deep With Your Online Adult Degree Programs

    I’ve had experiences both ways: some institutions go wide, others go deep. For those that go wide, I’ve often seen siloed marketing efforts, inefficient allocations of resources, and sporadic and unpredictable enrollment. For those that go deep, I see the following benefits: 

    More Students Attracted

    Broadened appeal for students already interested in the primary program: By offering more concentrations within a well-established program, or adjacent degrees within the same field, your institution can appeal to a broader range of interests and career goals within your current student audience base.

    More options for prospective students due to increased specialization: Specialized degrees and concentrations allow students to tailor their education to their specific interests and career paths, making the program more attractive to applicants seeking focused expertise.

    Increased Marketing Efficiency

    Ability to leverage existing web pages and SEO for the main program: Concentration pages can be added as subpages to the main program’s page, which likely already has a strong search engine optimization (SEO) presence. This setup benefits from the existing search engine rankings and requires less effort than starting marketing from scratch for a new program.

    Faster path to high search rankings for new concentrations, creating a marketing loop: The SEO efforts for the main program boost the visibility of the new concentrations, which in turn contribute to the overall authority and ranking of the main program’s page. This synergy creates a self-reinforcing cycle that enhances the visibility of all offerings.

    Enhanced paid marketing efficiencies: Adding concentrations in areas where significant traffic already exists for broad terms — like “MBA,” “business degree,” or “finance degree” for an MBA program — allows institutions to more effectively utilize their paid advertising budgets. Expanding the program options for your existing traffic allows you to improve your click-to-lead conversion rates, increase your number of leads, and enhance your downstream successes in areas such as enrollments and completions. This approach allows for a more efficient use of marketing investments, providing more options for prospective students within the same budget.

    Faster Accreditation Process

    Streamlined accreditation process by expanding within an already accredited program: Adding concentrations within an existing program simplifies the accreditation process. Because the core program is already accredited, expanding it with concentrations requires fewer approvals and less bureaucracy than launching an entirely new program.

    Ready to Go Deep With One of Your Online Adult Degree Programs?

    If you’ve seen success with an online adult degree program offering, you’ve already taken a momentous step toward growth — which is something to be proud of. It also creates massive opportunity, and Archer Education is poised to help you capitalize on it. 

    Archer is different from other agencies. We work as your online growth enablement partner, helping you to foster self-sufficiency over the long haul through collaboration, storytelling, and cutting-edge student engagement technology. 

    We’ve helped dozens of institutions increase enrollment and retention through a going deep approach, and your institution could be next. And once you’ve solidified the reputation and success of your core online offering by going deep, we’ll be ready to help you pivot to a wider approach to expand your position in online learning.

    Contact us today to learn more about what Archer can do for you. 

    Subscribe to the Higher Ed Marketing Journal:

    Source link

  • AI in Practice: Using ChatGPT to Create a Training Program

    AI in Practice: Using ChatGPT to Create a Training Program

    by Julie Burrell | September 24, 2024

    Like many HR professionals, Colorado Community College System’s Jennifer Parker was grappling with an increase in incivility on campus. She set about creating a civility training program that would be convenient and interactive. However, she faced a considerable hurdle: the challenges of creating a virtual training program from scratch, solo. Parker’s creative answer to one of these challenges — writing scripts for her under-10-minute videos — was to put ChatGPT to work for her. 

    How did she do it? This excerpt from her article, A Kinder Campus: Building an AI-Powered, Repeatable and Fun Civility Training Program, offers several tips.

    Using ChatGPT for Training and Professional Development

    I love using ChatGPT. It is such a great tool. Let me say that again: it’s such a great tool. I look at ChatGPT as a brainstorming partner. I don’t use it to write my scripts, but I do use it to get me started or to fix what I’ve written. I ask questions that I already know the answer to. I’m not using it for technical guidance in any way.

    What should you consider when you use ChatGPT for scriptwriting and training sessions?

    1. Make ChatGPT an expert. In my prompts, I often use the phrase, “Act like a subject matter expert on [a topic].” This helps define both the need and the audience for the information. If I’m looking for a list of reasons why people are uncivil on college campuses, I might prompt with, “Act like an HR director of a college campus and give me a list of ways employees are acting uncivil in the workplace.” Using the phrase above gives parameters on the types of answers ChatGPT will offer, as well as shape the perspective of the answers as for and about higher ed HR.
    2. Be specific about what you’re looking for. “I’m creating a training on active listening. This is for employees on a college campus. Create three scenarios in a classroom or office setting of employees acting unkind to each other. Also provide two solutions to those scenarios using active listening. Then, create a list of action steps I can use to teach employees how to actively listen based on these scenarios.” Being as specific as possible can help get you where you want to go. Once I get answers from ChatGPT, I can then decide if I need to change direction, start over or just get more ideas. There is no wrong step. It’s just you and your partner figuring things out.
    3. Sometimes ChatGPT can get stuck in a rut. It will start giving you the same or similar answers no matter how you reword things. My solution is to start a new conversation. I also change the prompt. Don’t be afraid to play around, to ask a million questions, or even tell ChatGPT it’s wrong. I often type something like, “That’s not what I’m looking for. You gave me a list of______, but what I need is ______. Please try again.” This helps the system to reset.
    4. Once I get close to what I want, I paste it all in another document, rewrite, and cite my sources. I use this document as an outline to rewrite it all in my own voice. I make sure it sounds like how I talk and write. This is key. No one wants to listen to ChatGPT’s voice. And I guarantee that people will know if you’re using its voice — it has a very conspicuous style. Once I’ve honed my script, I ensure that I find relevant sources to back the information up and cite the sources at the end of my documents, just in case I need to refer to them.

    What you’ll see here is an example of how I used ChatGPT to help me write the scripts for the micro-session on conflict. It’s an iterative but replicable process. I knew what the session would cover, but I wanted to brainstorm with ChatGPT.

    Once I’ve had multiple conversations with the chatbot, I go back through the entire script and pick out what I want to use. I make sure it’s in my own voice and then I’m ready to record. I also used ChatGPT to help with creating the activities and discussion questions in the rest of the micro-session.

    I know using ChatGPT can feel overwhelming but rest assured that you can’t really make a mistake. (And if you’re worried the machines are going to take over, throw in a “Thank you!” or “You’re awesome!” occasionally for appeasement’s sake.)

    About the author: Jennifer Parker is assistant director of HR operations at the Colorado Community College System.

    More Resources

    • Read Parker’s full article on creating a civility training program with help from AI.
    • Learn more about ChatGPT and other chatbots.
    • Explore CUPA-HR’s Civility in the Workplace Toolkit.



    Source link

  • Toward a Sector-Wide AI Tutor R&D Program –

    Toward a Sector-Wide AI Tutor R&D Program –

    EdTech seems to go through perpetual cycles of infatuation and disappointment with some new version of a personalized one-on-one tutor available to every learner everywhere. The recent strides in generative AI give me hope that the goal may finally be within reach this time. That said, I see the same sloppiness that marred so many EdTech infatuation moments. The concrete is being poured on educational applications that use a very powerful yet inherently unpredictable technology in education. We will build on a faulty foundation if we get it wrong now.

    I’ve seen this happen countless times before, both with individual applications and with entire application categories. For example, one reason we don’t get a lot of good data from publisher courseware and homework platforms is that many of them were simply not designed with learning analytics in mind. As hard as that is to believe, the last question we seem to ask when building a new EdTech application is “How will we know if it works?” Having failed to consider that question when building the early versions of their applications, publishers have had a difficult time solving for it later.

    In this post, I propose a programmatic, sector-wide approach to the challenge of building a solid foundation for AI tutors, balancing needs for speed, scalability, and safety.

    The temptation

    Before we get to the details, it’s worth considering why the idea of an AI tutor can be so alluring. I have always believed that education is primal. It’s hard-wired into humans. Not just learning but teaching. Our species should have been called homo docens. In a recent keynote on AI and durable skills, I argued that our tendency to teach and learn from each other through communications and transportation technologies formed the engine of human civilization’s advancement. That’s why so many of us have a memory of a great teacher who had a huge impact on our lives. It’s why the best longitudinal study we have, conducted by Gallup and Purdue University, provides empirical evidence that having one college professor who made us excited about learning can improve our lives across a wide range of outcomes, from economic prosperity to physical and mental health to our social lives. And it’s probably why the Khans’ video gives me chills:

    Check your own emotions right now. Did you have a visceral reaction to the video? I did.

    Unfortunately, one small demonstration does not prove we have reached the goal. The Khanmingo AI tutor pilot has uncovered a number of problems, including factual errors like incorrect math and flawed tutoring. (Kudos to Khan Academy for being open about their state of progress by the way.)

    We have not yet achieved that magical robot tutor. How do we get there? And how will we know that we’ve arrived?

    Start with data scientists, but don’t stop there

    As I read some of the early literature, I see an all-too-familiar pattern: technologists build the platforms, data scientists decide which data are important to capture, and they consult learning designers and researchers. However, all too often, the research design clearly originates from a technologist’s perspective, showing relatively little knowledge of detailed learning science methods or findings. A good example of this mindset’s strengths and weaknesses is Google’s recent paper, “Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach“. It reads like a paper largely concieved by technologists who work on improving generative AI and sharpened up by educational research specialists they consulted with after they already had the research project largely defined.

    The paper proposes evaluation rubrics for five dimensions of generative AI tutors:

    • Clarity and Accuracy of Responses: This dimension evaluates how well the AI tutor delivers clear, correct, and understandable responses. The focus is on ensuring that the information provided by the AI is accurate and easy for students to comprehend. High clarity and accuracy are critical for effective learning and avoiding the spread of misinformation.
    • Contextual Relevance and Adaptivity: This dimension assesses the AI’s ability to provide responses that are contextually appropriate and adapt to the specific needs of each student. It includes the AI’s capability to tailor its guidance based on the student’s current understanding and the specific learning context. Adaptive learning helps in personalizing the educational experience, making it more relevant and engaging for each learner.
    • Engagement and Motivation: This dimension measures how effectively the AI tutor can engage and motivate students. It looks at the AI’s ability to maintain students’ interest and encourage their participation in the learning process. Engaging and motivating students is essential for sustained learning and for fostering a positive educational environment.
    • Error Handling and Feedback Quality: This dimension evaluates how well the AI handles errors and provides feedback. It examines the AI’s ability to recognize when a student makes a mistake and to offer constructive feedback that helps the student understand and learn from their errors. High-quality error handling and feedback are crucial for effective learning, as they guide students towards the correct understanding and improvement.
    • Ethical Considerations and Bias Mitigation: This dimension focuses on the ethical implications of using AI in education and the measures taken to mitigate bias. It includes evaluating how the AI handles sensitive topics, ensures fairness, and respects student privacy. Addressing ethical considerations and mitigating bias are vital to ensure that the AI supports equitable learning opportunities for all students.

    Of these, the paper provides clear rubrics for the first four and is a little less concrete on the fifth. Notice, though, that most of these are similar dimensions that generative AI companies use to evaluate their products generically. That’s not bad. On the contrary, establishing standardized, education-specific rubrics with high inter-rater reliability across these five dimensions is the first component of the programmatic, sector-wide approach to AI tutors that we need. Notice these are all qualitative assessments. That’s not bad but, for example, we do have quantitative data available on error handling in the form of feedback and hints (which I’ll delve into momentarily).

    That said, the paper lacks many critical research components, particularly regarding the LearnLM-Tutor software the researchers were testing. Let’s start with the authors not providing outcomes data anywhere in the 50-page paper. Did LearnLM-Tutor improve student outcomes? Make them worse? Have no effect? Work better in some contexts than others? We don’t know.

    We also don’t know how LearnLM-Tutor incorporates learning science. For example, on the question of cognitive load, the authors write,

    We designed LearnLM Tutor to manage cognitive load by breaking down complex tasks into smaller, manageable components and providing scaffolded support through hints and feedback. The goal is to maintain an optimal balance between intrinsic, extraneous, and germane cognitive load.

    Towards Responsible Development ofGenerative AI for Education: An Evaluation-Driven Approach

    How, specifically, did they do this? What measures did they take? What relevant behaviors were they able to elicit from their LLM-based tutor? How are those behaviors grounded in specific research findings about cognitive load? How closely do they reproduce the principals that produced the research findings they’re drawing from? And did it work?

    We don’t know.

    The authors are also vague about Intelligent Tutoring Systems (ITS) research. They write,

    Systematic reviews and meta-analyses have shown that intelligent tutoring systems (ITS) can significantly improve student learning outcomes. For example, Kulik and Fletcher’s meta-analytic review demonstrates that ITS can lead to substantial improvements in learning compared to traditional instructional methods.

    Towards Responsible Development ofGenerative AI for Education: An Evaluation-Driven Approach

    That body of research was conducted over a relatively small number of ITS implementations because a relatively small number of these systems exist and have published research behind them. Further, the research often cites specific characteristics of these tutoring systems that lead to positive outcomes, with supporting data. Which of these characteristics does LearnLM Tutor support? Why do we have reason to believe that Google’s system will achieve the same results?

    We don’t know.

    I’m being a little unfair to the authors by critiquing the paper for what it isn’t about. Its qualitative, AI-aligned assessments are real contributions. They are necessary for a programmatic, sector-wide approach to AI tutor development. They simply are not sufficient.

    ITS data sets for fine-tuning

    ITS research is a good place to start if we’re looking to anchor our AI tutor improvement and testing program in solid research with data sets and experimental protocols that we can re-use and adapt. The first step is to explore how we can utilize the existing body of work to improve AI tutors today. The end goal is to develop standards for integrating the ongoing ITS research (and other data-backed research streams) into continuous improvement of AI tutors.

    One key short-term opportunity is hints and feedback. If, for the moment, we stick with the notion of a “tutor” as software engaging in adaptive, turn-based coaching of students on solving homework problems, then hints and feedback are core to the tutor’s function. ITS research has produced high-quality, publicly available data sets with good findings on these elements. The sector should construct, test, and refine an LLM fine-tuning data set on hints and feedback. This work must include developing standards for data preprocessing, quality assurance, and ethical use. These are non-trivial but achievable goals.

    The hints and feedback work could form a beachhead. It would help us identify gaps in existing research, challenges in using ITS data this way, and the effectiveness of fine-tuning. For example, I’d be interested in seeing whether the experimental designs used in hints and feedback ITS research papers could be replicated with an LLM that has been fine-tuned using the research data. In the process, we want to adopt and standardize protocols for preserving student privacy, protecting author rights, and other concerns that are generally taken into account in high-quality IRB-approved studies. These practices should be baked into the technology itself when possible and supported by evaluation rubrics when it is not.

    While this foundational work is being undertaken, the ITS research community could review its other findings and data sets to see which additional research data sets could be harnessed to improve LLM tutors and develop a research agenda that strengthens the bridge being built between that research and LLM tutoring.

    The larger limitations of this approach will likely spring the uneven and relatively sparse coverage of course subjects, designs, and student populations. We can learn a lot about developing a strategy for uses these sorts of data from ITS research. But to achieve the breadth and depth of data required, we’ll need to augment this body of work with another approach that can scale quickly.

    Expanding data sets through interoperability

    Hints and feedback are great examples of a massive missed opportunity cost. Virtually all LMSs, courseware, and homework platforms support feedback. Many also support hints. Combined, these systems represent a massive opportunity to gather data about usage and effectiveness of hints and feedback across a wide range of subjects and contexts. We already know how the relevant data need to be represented for research purposes because we have examples from ITS implementations. Note that these data include both design elements—like the assessment question, the hints, the feedback, and annotations about the pedagogical intent—and student performance when they use the hints and feedback. So if, for example, we were looking at 1EdTech standards, we would need to expand both Common Cartridge and Caliper standards to incorporate these elements.

    This approach offers several benefits. First, we would gain access to massive cross-platform data sets that could be used to fine-tune AI models. Second, these standards would enable scaled platforms like LMSs to support proven metheds for testing the quality of hints and feedback elements. Doing so would provide benefit to students using today’s platforms while enabling improvement of the training data sets for AI tutors. The data would be extremely messy, especially at first. But the interoperability would enable a virtuous cycle of continuous improvement.

    The influence of interoperability standards on shaping EdTech is often underestimated and misunderstood. !EdTech was first created when publishers realized they needed a way to get their content into new teaching systems that were then called Instructional Management Systems (IMS). Common Cartridge was the first standard created by the organization now known as 1EdTech. Later, Common Cartridge export made migration from one LMS to another much more feasible, thus aiding in breaking the product category out of what was then a virtual monopoly. And I would guess that perhaps 30% or more of the start-ups at the annual ASU+GSV conference would not exist if they could not integrate with the LMS via the Learning Tool Interoperability (LTI) standard. Interoperability is a vector for accelerating change. Creating interoperabiltiy around hints and feedback—including both the importing of them into learning systems and passing student performance impact data—could accelerate the adoption of effective interactive tutoring responses, whether they are delivered by AI or more traditional means.

    Again, hints and feedback are the beachhead, not the end game. Ultimately, we want to capture high-quality training data across a broad range of contexts on the full spectrum of pedagogical approaches.

    Capturing learning design

    If we widen the view beyond the narrow goal of good turn-taking tutorial responses, we really want our AI to understand the full scope of pedagogical intent and which pedagogical moves have the desired effect (to the degree the latter is measurable). Another simple example of a construct we often want to capture in relation to the full design is the learning objective. ChatGPT has a reasonably good native understanding of learning objectives, how to craft them, and how they relate to gross elements of a learning design like assessments. It could improve significantly if it were trained on annotated data. Further, developing annotations for a broad spectrum of course design elements could improve its tutoring output substantially. For example, well-designed incorrect answers to questions (or “distractors”) often test for misconceptions regarding a learning objective. If distractors in a training set were specifically tagged as such, the AI could better learn to identify and probe for misconceptions. This is a subtle and difficult skill even for human experts but it is also a critical capability for a tutor (whether human or otherwise).

    This is one of several reasons why I believe focusing effort on developing AI learning design assistants supporting current-generation learning platforms is advantageous. We can capture a rich array of learning design moves at design time. Some of these we already know how to capture through decades of ITS design. Others are almost completely dark. We have very little data on design intent and even less on the impact of specific design elements on achieving the intended learning goals. I’m in the very early stages of exploring this problem now. Despite having decades of experience in the field, I am astonished at the variability in learning design approaches, much of which is motivated and little of which is tested (or even known within individual institutions).

    On the other side, at-scale platforms like LMSs have implemented many features in common that are not captured in today’s interoperability standards. For example, every LMS I know of implements learning objectives and has some means of linking them to activities. Implementation details may vary. But we are nowhere close to capturing even the least-common-denominator functionality. Importantly, many of these functions are not widely used because of the labor involved. While LMSs can link learning objectives to learning activities, many course builders don’t do it. If an AI could help capture these learning design relationships, and if it could export content to a learning platform in a standard format that preserves those elements, we would have the foundations for more useful learning analytics, including learning design efficacy analytics. Those analytics, in turn, could drive improvement of the course designs, creating a virtuous cycle. These data could then be exported for model training (with proper privacy controls and author permissions, of course). Meanwhile, less common features such as flagging a distractor as testing for a misconception could be included as optional elements, creating positive pressure to improve both the quality of the learning experiences delivered in current-generation systems and the quality of the data sets for training AI.

    Working at design time also puts a human in the loop. Let’s say our workflow follows these steps:

    1. The AI is prompted to conduct turn-taking design interviews of human experts, following a protocol intended to capture all the important design elements.
    2. The AI generates a draft of the learning design. Behind the scenes, the design elements are both shaped by and associated with the metadata schemas from the interoperability standards.
    3. The human experts edit the design. These edits are captured, along with annotations regarding the reasons for the edits. (Think Word or Google Docs with comments.) This becomes one data set that can be used to further fine-tune the model, either generally or for specific populations and contexts.
    4. The designs are exported using the interoperability standards into production learning platforms. The complementary learning efficacy analytics standards provide telemetry on the student behavior and performance within a given design. This becomes another data set that could potentially be used for improving the model.
    5. The human learning designers improve the course designs based on the standards-enabled telemetry. They test the revised course designs for efficacy. This becomes yet another potential data set. Given this final set in the chain, we can look at designer input into the model, the model’s output, the changes human designers made, and improved iterations of the original design—all either aggregated across populations and contexts or focused on a specific population and context.

    This can be accomplished using the learning platforms that exist today, at scale. Humans would always supervise and revise the content before it reaches the students, and humans would decide which data they would share under what conditions for the purposes of model tuning. The use of the data and the pace of movement toward student-facing AI become policy-driven decisions rather than technology-driven. At each of the steps above, humans make decisions. The process allows for control and visibility regarding the plethora of ethical challenges that face integrating AI into education. Among other things, this workflow creates a policy laboratory.

    This approach doesn’t rule out simultaneously testing and using student-facing AI immediately. Again, that becomes a question of policy.

    Conclusion

    My intention here has been to outline a suite of “shovel-ready” initiatives that could be implemented realitvely quickly at scale. It is not comprehensive; nor does it attempt to even touch the rich range of critical research projects that are more investigational. On the contrary, the approach I outline here should open up a lot of new territory for both research and implementation while ensuring the concrete already being poured results in a safe, reliable, science- and policy-driven foundation.

    We can’t just sit by and let AI happen to us and our students. Nor can we let technologists and corporations become the primary drivers of the direction we take. While I’ve seen many policy white papers and AI ethics rubrics being produced, our approach to understanding the potential and mitigating the risks of EdTech AI in general and EdTech tutors in particular is moving at a snail’s pace relative to product development and implementation. We have to implement a broad, coordinated response.

    Now.

    Source link