Curriculum assessment and evaluation projects may not always spark immediate enthusiasm, but a recent well-organized team project in our higher education program demonstrated the power of collaboration. Thanks to the incredible participation of our faculty, improvements in student learning were not just planned, but successfully implemented. The changes made as part of this academic assessment/evaluation project to enhance student success yielded significant improvements. The positive impact on student learning results is rewarding when the impact on student learning results is positive!
The measures are crucial to our curriculum assessment/evaluation, program accreditation, and appraising evidence of student success. A key aspect of our approach is the use of both indirect and direct measures to gauge student learning within program and course assessments. This method allows us to redesign course assessments, incorporating authentic direct and indirect measures to foster student success.
What is a Direct Measure Assessment?
Our program faculty evaluated and updated our program curriculum, including assessment measures, as an aspect of continuous quality improvement within our educational assessment program. A direct measure objectively quantifies student learning (Bugeja & Garrett, 2019). It is crucial that these direct measures, which include course assignments (for example, papers, demonstrations, worksheets, presentations, virtual simulations, live simulations), examinations, capstone projects, student portfolios, and internship evaluations (Bugeja & Garrett, 2019), align with learning outcomes. This alignment ensures that the assessment accurately reflects the intended learning goals.
Well-constructed rubrics further enhance the quality of the assessment measure by communicating the criteria to the student and increasing the objectivity of the measure (Postmes et al., 2023). Linking examinations to learning outcomes and blueprints (List of topics the examination is assessing) increases authenticity (Abdellatif & Al-Shahrani, 2019). See the sample alignment template included at the end of this article.
What is an Indirect Measure Assessment?
An indirect measure is a subjective perception of the experience (Bugeja & Garrett, 2019). Examples of indirect measures include surveys (student, employer), focus groups, and exit interviews (Bugeja & Garrett, 2019). Indirect measures should not solely measure a course outcome or competency but should support a direct assessment measure. For example, in a major course assignment or project such as a Change Theory Application Project paper or poster presentation (direct measure), the student also completes a reflective self-evaluation survey of the project specific to learning outcomes or competencies (indirect measure). The indirect measure is valuable because it provides a different perspective but supports the direct measure.
Implementing Direct and Indirect Assessments
The program-level outcomes measure student success at program graduation. They are also known as end-of-program outcomes or terminal program outcomes. Course-level outcomes should match the attainment of the program-level outcomes.
After reviewing program-level assessment plans, the focus becomes the course level. We recommend reviewing and updating an existing alignment table or creating one that includes the course outcomes, module/unit objectives, competencies, learning activities, and assessments (modify the items per your school). We have included a sample alignment table at the end of this article. The alignment table map helps check the Revised Bloom’s Taxonomy Leveling and ensures all course outcomes and competencies link to course assessments (direct and indirect measures).
Next, examine the current course assessments. Identify course assessments worthy of updating and revision or develop a new assessment measure. Use mostly direct assessments with a few supporting indirect assessments, such as exit evaluations, focus groups, alums, and employer surveys. Remember, an indirect assessment should only add another viewpoint that supports the direct assessment measure.
Map the course outcomes, module learning objectives, course-assigned knowledge, skills, attitudes/values, competencies, and learning activities (for example, readings, videos, posters) in an alignment table. Add the final assessments to the alignment table after updating course assessment measures. Check that most measures are direct and that all course outcomes, objectives, and competencies link to a direct measure. Use only a few supporting indirect measures to safeguard genuine assessment of learning.
Summary
Educational assessment is valuable for informing curricula, demonstrating student learning, and determining program effectiveness for academic accreditation. Using direct and indirect measures in education provides data about student learning and program evaluation. Alignment mapping can help plan and check for consistency in the course. Multiple data measures, such as direct measures and a few supporting indirect measures, increase confidence in the findings and analysis, known as triangulation (Demeter et al., 2019). After completing course revisions with a particular emphasis on using direct and indirect assessment measures, we have noted improvements in student learning assessment at the course level.
Course Level Alignment Table Map
Course Outcomes
Module Objectives
Course Competencies
Learning Activities
Course Learning Assessments (Identify Direct and Indirect Measures)
Course Level Alignment Table Map Tips:
Add and delete rows and columns as needed.
The verb levels in the outcomes, objectives, and competencies should align with the learning activities and assessment measures.
Not all courses will have competencies; this may vary depending on the discipline.
Colleges and universities vary in the word use of outcomes and objectives, so modify the headings in the alignment table to match your school.
Nancyruth Leibold, EdD, RN, MSN, PHN, CNE, AHN-BC is an Associate Professor of Nursing at Minnesota State University, Mankato. Nancyruth is the Health Sciences Editor at Multimedia Educational Resource for Learning and Online Teaching (MERLOT).
Laura Schwarz, DNP, RN, PHN, CNE, AHN-BC is a Professor at Minnesota State University, Mankato. Laura is a Certified Peer Reviewer for Quality Matters.
References
Abdellatif, H., & Al-Shahrani, A. M. (2019). Effect of blueprinting methods on test difficulty, discrimination, and reliability indices: cross-sectional study in an integrated learning program. Advances in Medical Education and Practice, 10, 23–30. https://doi.org/10.2147/AMEP.S190827
Bugeja, M., & Garrett, M. (2019). “Making the connection”: Aggregate internship data as direct and indirect measure informing curricula and assessment. Journalism & Mass Communication Educator, 74(1), 17–30. https://doi.org/10.1177/1077695817749077
Demeter, E., Robinson, C., & Frederick, J. G. (2019). Holistically assessing critical thinking and written communication learning outcomes with direct and indirect measures. Research & Practice in Assessment, 14(1), 41-51.
Postmes, L., Bouwmeester, R., de Kleijn, R., & van der Schaaf, M. (2023). Supervisors’ untrained postgraduate rubric use for formative and summative purposes. Assessment and Evaluation in Higher Education, 48(1), 41–55. https://doi.org/10.1080/02602938.2021.2021390
FARMINGTON HILLS, Mich. – Gale, part of Cengage Group, is pleased to announce that it has received four platinum awards in LibraryWorks’ eleventh annual Modern Library Awards (MLAs) for its adult learning and youth large print resources. The MLAs were created to recognize the top products and services in the library industry in a truly unbiased format. This is the fourth consecutive year Gale has been recognized by the MLAs for product excellence.
“Being recognized for the fourth year running is a testament to Gale’s unwavering commitment to innovative learning resources,” said Paul Gazzolo, senior vice president and general manager at Gale. “Customer motivation is at the heart of everything we do to provide the best accessible learning and reading experiences for learners of all ages. This recognition is more validation that our products continue to break barriers and reach new heights in education, and we are thrilled to receive such an honor.”
Products and services were submitted to LibraryWorks in the fall of 2024 and sent to more than 90,000 librarians at public, K-12, academic and special libraries. Only customers experienced with these products/services in their facilities were permitted to judge them, resulting in a truly unbiased score.
Each judge scored the products on a numeric basis from 1-10, based on a series of questions regarding functionality, value, customer service experience and overall satisfaction.
Gale’s 2025 MLA platinum award winners include:
Gale Presents: Peterson’s Test and Career Prep(platinum award): offers a comprehensive set of tools to support individuals on their educational and professional journeys. Whether for high school students preparing for standardized tests, working adults seeking career changes, or those pursuing further education, the platform provides resources for success.
“This is the best database relating to college and career preparation for teens that we use at our library! It has so many resources and the patrons love it!” –MLA judge
Gale Presents: Excel Adult High School(platinum award): A 21.5-credit, self-paced online high school completion program that allows public libraries to offer accredited high school diplomas to adults who wish to prepare for entry into the workforce, further their careers, or attend college.
“Our patrons have overcome significant challenges to achieve their goal of earning a high school diploma through the Gale Presents: Excel Adult High School. I cannot emphasize enough just how impactful this program is and how it improves the lives of our graduates. Gale has helped our library system further our mission to provide services that have a truly positive impact on our community.” –MLA judge
Thorndike Press – Youth Large Print(platinum award): A viable reading intervention tool, similar in size as standard print with the same cover art. These books have been proven to improve decoding, increase comprehension, and reduce anxiety in developing, below-grade-level, and emerging bilingual readers.
“Thorndike Large Print books for youth have been instrumental in helping me reach students who are struggling with reading, students who don’t enjoy reading and students who have no issues with reading! These books are for ALL my students, and because they are so accessible, ALL students can enjoy them and reap the benefits. I couldn’t highly recommend them enough.” –MLA judge
Gale Presents: Udemy (platinum award): Public libraries can connect patrons to nearly 30,000 on-demand video courses for upskilling or reskilling in business, technology, and personal development. Thousands of courses are taught by native speakers in 14 different languages, helping libraries reach their diverse communities.
“(Our) Public Library is proud to offer Udemy Business as a valuable resource for our community. It provides access to a wide range of professional development courses. It helps equip our patrons with educational tools so that they are able to be successful in their career and beyond.”–MLA judge
Jenny Newman, publisher and MLA program manager noted, “It’s hardly a surprise that Gale continues to score high. The strong partnerships they have established with libraries are what differentiate and drive their industry leadership. The quality and innovation of their adult learning and youth large print resources are truly impacting and changing the lives of all learners, keeping the company at the forefront of the market.”
About Cengage Group and Gale
Cengage Group, an education technology company serving millions of learners in 165 countries, advances the way students learn through quality, digital experiences. The company currently serves the K-12, higher education, professional, library, English language teaching and workforce training markets worldwide. Gale, part of Cengage Group, provides libraries with original and curated content, as well as the modern research tools and technology that are crucial in connecting libraries to learning, and learners to libraries. For more than 65 years, Gale has partnered with libraries around the world to empower the discovery of knowledge and insights – where, when and how people need it. Gale has 500 employees globally with its main operations in Farmington Hills, Michigan. For more information, please visit www.gale.com.
Follow Gale on:
About LibraryWorks
LibraryWorks helps administrators to make informed decisions about library technology, automation and software, collection development and management, facilities and furnishings, staffing, purchasing, and other areas that drive effective strategic planning and day-to-day operations. Our family of resources can enable you to identify best practices, monitor trends, evaluate new products and services, apply for grants and funding, post or find a job, and even enjoy some library humor. https://www.libraryworks.com/
About the Modern Library Awards program The Modern Library Awards (MLAs) is a review program designed to recognize elite products and services in the market which can help library management personnel enhance the quality-of-experience for the library user and increase the performance of their library systems. https://www.modernlibraryawards.com/
eSchool Media staff cover education technology in all its aspects–from legislation and litigation, to best practices, to lessons learned and new products. First published in March of 1998 as a monthly print and digital newspaper, eSchool Media provides the news and information necessary to help K-20 decision-makers successfully use technology and innovation to transform schools and colleges and achieve their educational goals.
One highlight from FETC’s Startup Pavilion is Florida-based Scholar Education, which uses AI chatbot dogs to help tutor students and give feedback to teachers. How it works: A friendly AI-powered classroom assistant provides academic guidance and encourages engagement. The AI dogs will deliver daily reports to parents so they can see feedback on their kids’ learning, creating a direct line of communication between home and school. See it in action for yourself:
Kevin is a forward-thinking media executive with more than 25 years of experience building brands and audiences online, in print, and face to face. He is an acclaimed writer, editor, and commentator covering the intersection of society and technology, especially education technology. You can reach Kevin at [email protected]
The series, premiering on February 5, was filmed at Bell English Schools’s Cambridge premisis in August 2025, which hosted Dong-Il Sung, Kwang-Kyu Kim, Ki-Jun Um, Hyuk Jang, and Seung-Hwan Shin for two weeks in August 2024.
The program, Shala Shala, follows the actors’ authentic experiences of learning English alongside other students, taking part in activities, and staying with host families and in Bell Cambridge’s on-campus residences.
“These are male, middle-aged actors who are famous in South Korea, but who have got to a certain point in their career and maybe they want to try more roles in English or maybe they want to travel personally with their families,” Rebecca Stead, head of marketing at Bell Educational Services told The PIE News.
“So, the premise of the show was that it’s never too late to late to learn English,” Stead added.
To make it as authentic as possible, 50 hidden cameras were set up around the school to capture the actors’ true experiences taking classes with Bell’s summer learners.
While Stead maintained that the actors were “great students”, the program’s trailer reveals a somewhat bumpy path to language learning success.
“It was a big operation and really interesting for us staff to see how a TV program is made, and the other students were really excited to have the actors around,” she added.
The premise of the show was that it’s never too late to late to learn English
Rebecca Stead, Bell English
With seven schools for adult students and young learners across the UK, Bell’s Cambridge institution is its flagship location offering year-round language courses to learners of all ages.
“We’ve got these beautiful gardens and traditional buildings, and it’s in Cambridge, which is such an attractive destination. It’s very much that quintessential image of what a lot of people from other countries imagine the UK to be like,” Stead noted.
“Not only are we showcasing Bell but we’re also showcasing the UK and what a valuable experience it is to study and travel here, so hopefully it will be a positive thing for the industry as a whole.”
The program comes at a time of slowing recovery for the UK’s ELT sector, with levels likely to be a “new normal” for the sector, according to a recent report by English UK.
The program will be airing on YouTube as well as the South Korean television network JTBC on February 5, 2025.
This audio is auto-generated. Please let us know if you have feedback.
Schools across Southern California impacted by devastating wildfires this month are working to ensure students, families and staff are safe and have basic needs — all while attempting to restart instruction and as-normal-as-possible school routines after school closures.
At least 335 schools from Los Angeles, San Bernardino, Riverside, Ventura and San Diego counties had closed temporarily when fires broke out last week, affecting more than 211,000 students, according to the California Department of Education.
Two schools in the Los Angeles Unified School District — Palisades Charter Elementary and Marquez Charter Elementary — will need to be rebuilt due to fire damage, LAUSD said in a Jan. 13 statement.
At an event Tuesday in Washington, D.C., to highlight U.S. Department of Education initiatives under the Biden administration, Deputy Secretary of Education Cindy Marten, who previously served as superintendent of San Diego Unified School District, said the area is close-knit and that people have been “deeply affected” by the destructive wildfires.
“What we know is that precious schools have burned down and communities are reeling,” said Marten, adding that the U.S. Education Department will provide training and funding to communities affected by the disaster.
According to the California Department of Forestry and Fire Protection, known as CAL FIRE, 40,695 acres have burned and more than 12,300 structures have been destroyed. Several fires that started Jan. 7 or after still have not been fully contained.
Most schools in LAUSD — the second largest school system in the nation — reopened Monday after district employees cleaned schools and others worked “around the clock” over the weekend to ensure campuses were safe for students and staff, a Jan. 13 district statement said. By Wednesday, outdoor activities including P.E. and recess could resume at all campuses pending local conditions, and students at the two schools destroyed by fire were relocated to two other campuses, the district said.
“We have a unique opportunity to show the strength and resilience of our community in the face of adversity,” said Pamela Magee, executive director and principal of Palisades Charter High School, in a statement Jan 13. “By coming together, we can ensure that our students can stay in their learning environment, with their friends and mentors, at a time when they need it most.”
Schools in Malibu are closed through at least Jan. 21, while Santa Monica schools are open, according to the Santa Monica-Malibu Unified School District. The district and its partners have organized optional gathering spaces for children and teens displaced by the fires and not in school.
In the Pasadena Unified School District, more than 1,300 Pasadena USD staff members had homes within the burn zone, and the district is still determining the exact number of students and families impacted. That number is anticipated to be in the thousands, according to the California Department of Education.
The district is closed through Jan. 17, although students had access to optional, self-directed learning options, Superintendent Elizabeth Blanco wrote in a statement to the school district community Jan. 10.
“The health and safety of our PUSD community remain our highest priority as we navigate the significant impact of the fire on so many of our students, families, and staff,” said Blanco, adding that nearly half of the district’s employees live within the fire evacuation zone and that many staff, students and families lost their homes.
Odyssey Charter Schools, South Campus, in Altadena, California, and authorized by PUSD, was destroyed by the Eaton fire on Jan. 8. The 7-year-old school served about 375 students in grades TK-8.
“While our campus is closed, Odyssey Charter Schools South continues and will move forward stronger than ever. We’ve already built this school from an idea to a full institution. Then we rebuilt it again online during COVID and we built it a third time when we had to relocate so we are a resilient community and we already weathered many challenges,” said a video showing the fire’s destruction to the campus.
With the widespread impact of the wildfires and ongoing firefighting, the focus on learning is taking a backseat to supplying students, families and school employees with basic needs.
According to the California School Employees Association, Los Angeles Chapter 500, a union that represents school employees in LAUSD, the district is providing employees displaced from their primary residence up to five days paid time off, as well as $500 per displaced employee to help with short-term costs.
LAUSD has set up support hotlines for families and employees and posted resources for child care, food, transportation and housing assistance. Other area districts are doing the same.
To help children and their families impacted by the fires, the California Department of Education has partnered with SupplyBank.org to collect donations for school communities and their disaster relief efforts.
“In times of crisis, Californians consistently demonstrate their resilience and generosity as we continue to deal with the effects of climate change,” said State Superintendent of Public Instruction Tony Thurmond, in a Jan. 8 statement. “Let’s continue to unite and support those in need as they work to stay safe and rebuild.”
Los Angeles-area students are also being supported by charitable efforts from schools across the country. In Maryland, the Anne Arundel County Public Schools has created a donation drive titled #LiftLA.
“The rebuilding of Los Angeles will take years, if not decades,” said AACPS Superintendent of Schools Mark Bedell in a Jan. 16 statement. “Our efforts can make a difference for children we may never meet, but whose lives can be bettered by our generosity.”
In California, the fires have also led to adjustments with certain education requirements and deadlines. An executive order issued by Gov. Gavin Newsom on Jan. 14 addresses certain rules for early childhood education and K-12, including:
Waivers of regulations to allow childcare providers to accommodate additional children displaced by fires.
A directive for school districts to immediately enroll any student who is unhoused due to the fires.
The requirement that school systems continue to serve enrolled students who are displaced and are temporarily living outside their home district boundaries in the greater Los Angeles area.
The option for affected school systems to suspend adherence to a 180-day or 175-day school year calendar.
Additionally, the California Student Aid Commission has extended the March 3 priority state aid deadline to April 2 for high school students and California community college students in Los Angeles and Ventura counties seeking to attend or transfer to a four-year university.
Patsy Moskal and I have decided to be guest editors for Education Sciences for a special edition entitled,
“Trends in the Use of Generative Artificial Intelligence for Digital Learning.” (See below for a longer description.)
It is a most timely topic of deep interest to many in the academy. We would love to have you contribute an article for it. Your submission can be research, practitioner, or thought-based. It also does not have to be a long article (4,000-word minimum). Final articles will be due no later than July 1, 2025.
Feelings of belonging have a significant positive impact on academic success and progression, but we know that creating belonging isn’t as simple as putting up a welcome sign.
Belonging is not something that can be automatically created by an institution, regardless of its commitment to access and inclusion. To make students feel they belong in a higher education environment, having the power to shape and co-create the environments in which they participate is essential.
For students in higher education, liminal digital spaces (those informal areas of interaction that sit between formal academic environments and students’ broader social contexts) offer unique opportunities for students to lead, collaborate, learn and foster a sense of belonging, and the freedom to shape their learning environment and exercise agency in ways that may not be available within more formal institutional frameworks. They also offer opportunities for institutions to create places that nurture academic success without assuming responsibility for the development and delivery of all support.
But squaring the ownership, credibility and safeguarding triangle is complex, so how can universities do this while also embracing digital tools?
Taking ownership for learning
Focusing on digital spaces allows institutions to expand the space their students feel comfortable inhabiting and learning in, without limiting engagement from those who may not be free to meet at a specific time or be able to meet in person.
Digital learning resources can help students connect to their peers, further strengthening their sense of place within the institution. These spaces could act as connectors between university resource and student-driven exploration and learning in a way that more formal mechanisms sometimes fail to. At Manchester, resources such as My Learning Essentials (a blended skills support programme) can be used by the students within the spaces (via online resources) and signposted and recommended by peers (for scheduled support sessions).
Although this model exists elsewhere, at Manchester it is enhanced by the CATE-awarded Library Student Team, a group of current students who appreciate and often inhabit these spaces themselves. The combination of always available online, expert-led sessions and peer-led support means there is a multiplicity of avenues in the support. This allows the University to partner with, for example, its Students’ Union, and work alongside students and the wider institution by hosting these digital spaces, acting as mediators or facilitators, and ensuring the right balance of autonomy and support.
Keeping learning credible
Wider institutional support like My Learning Essentials already takes advantage of digital spaces by delivering both asynchronous online support and scheduled online sessions, and it can be easily integrated, signposted and shaped by the students using it.
These spaces need to be connected to the institution in such a way as to feel relevant and powerful. “Leaving” students to lead in spaces, giving them leadership responsibility without institutional support or backing, sets both them and these spaces up for failure.
Universities can work alongside students to help them define collective community values and principles, much like the community guidelines found in spaces like MYFest, a community-focused annual development event. Doing so ensures these liminal spaces are inclusive and responsive to the needs of all participants. Such spaces can also help students transition ‘out’ of the university environment and support others to build skills that they have already developed, such as by mentoring a student in a year below.
Safeguarding in a digital world
Universities should also allow students to follow the beat of their own drum and embrace digital outside of university spaces to further their learning.
Kai Prince, a PhD candidate in Maths at The University of Manchester, who runs a popular Discord server for fellow students, notes:
If the servers are led by a diverse group of students, I find that they’re also perfect for building a sense of belonging as students feel more comfortable in sharing their difficulties pseudo-anonymously and receiving peer-support, either by being informed on solutions or having their experiences, such as impostor syndrome, acknowledged.
Spaces like Discord allow students to engage in peer-led learning, but universities can enhance the quality of that learning by making available and investing in (as is done with My Learning Essentials) high-quality online materials, clear paths to wider support services and formal connections with societies or other academic groups. These mechanisms also help to keep the space within a student’s university experience, with all the expectations for behaviour and collegiality that entails.
The higher education sector is a complex and diverse space, welcoming new members to its communities each year. But it is often mired in a struggle to effectively engage and include each individual as a true part of the whole.
Work to address this needs to incorporate the students in spaces where the balance of power is tilted, by design, in their favour. Recognising the potential for digital spaces, for accessibility, support and familiarity for students as they enter higher education means that universities can put their efforts towards connecting, but not dictating, the direction of students and helping them forge their own learning journeys as part of the wider university community.
By Michelle Morgan, Dean of Students at the University of East London.
In the UK, we have a well-established education system across different levels of learning including primary, secondary, further and higher education. For each level, there is a comprehensive structure that is regulated and monitored alongside extensive information. However, at present, they generally function in isolation.
The Government’s recent Curriculum and Assessment Review has asked for suggestions to improve the curriculum and assessment system for the 16-19 year study group. This group includes a range of qualifications including GCSEs, A-levels, BTECs, T Levels and apprenticeships. The main purpose of the Review is to
ensure that the curriculum balances ambition, relevance, flexibility and inclusivity for all children and young people.
However, as part of this review, could it also look at how the different levels of study build on one another? Could the sectors come together and use their extensive knowledge for their level and type of study, to create an integrated road map across secondary, further and higher education where skills, knowledge, competencies and attributes (and how they translate into employability skills) are clearly articulated? We could call this a National Learning Framework. It could align with the learning gain programme led by the Office for Students (OfS).
The benefits of a National Learning Framework
There would be a number of benefits to adopting this approach:
It would provide a clear resource for all stakeholders, including students and staff in educational organisations, policymakers, Government bodies, Regulators and Quality Standard bodies (such as Ofsted, the Office for Students and QAA) and business and industry. It would also help manage the general public perception of higher education.
This approach would join up the regulatory bodies responsible for the different sectors. It would help create a collaborative, consistent learning and teaching approach, by setting and explaining the aims and objectives of the various types of education providers.
It would explain and articulate the differences in learning, teaching and assessment approaches across the array of secondary and further education qualifications that are available and used as progression qualifications into higher education. For example, A-Levels are mainly taught in schools and assessed by end-of-year exams. ‘Other’ qualifications such as BTEC, Access and Other Level 3 qualifications taught in college have more diverse assessments.
It would help universities more effectively bridge the learning and experience transition into higher education across all entry qualifications. We know students from the ‘Other’ qualification groups are often from disadvantaged backgrounds, which can affect retention, progression and success at university as research highlights (see also this NEON report). Students with other qualifications are more likely to withdraw than those with A-Levels. However, as this recent reportPrior learning experience, study expectations of A-Level and BTEC students on entry to universityhighlights, it is not the BTEC qualification per se that is the problem but the transition support into university study that needs improvement.
It would also address assumptions about how learning occurs at each level of study. For example, because young people use media technology to live and socialise, it is assumed the same is the case with learning. Accessing teaching and learning material, especially in schools, remains largely traditional: the main sources of information are course textbooks and handwritten notes, although since the Covid-19 Pandemic, the use of coursework submission and basic virtual learning environments (VLEs) is on the increase.
If we clearly communicate to students the learning that occurs throughout each level of their study, and what skills, knowledge, competencies and attributes they should obtain as a result, this can help with their confidence levels and their employability opportunities as they can better articulate what they have achieved.
What could an integrated learning approach across all levels of study via a NationalLearning Framework look like?
The Employability Skills Pyramid created for levels 4 to 7 in higher education with colleagues in a previous university where I worked could be extended to include Levels 2/3 and apprenticeships to create a National Learning Framework. The language used to construct the knowledge, skills and attribute grids used by course leaders purposely integrated the QAA statements for degrees (see accompanying document Appendix 1) .
By adding Levels 2 and 3, including apprenticeship qualifications and articulating the differences between each qualification, the education sector could understand what is achieved within and between different levels of study and qualifications (see Figure 1).
Key stakeholders could come together from across all levels of study to map out and agree on the language to adopt for consistency across the various levels and qualifications.
Integrated National Learning Framework across Secondary, Further andHigher Education
Alongside the National Learning Framework, a common transition approach drawing on the same definitions across all levels of study would be valuable. Students and staff could gain the understanding required to foster successful transitions between phases. An example is provided below.
Supporting transitions across the National Learning Framework using similar terminology
The Student Experience Transitions (SET) Model was designed to support courses of various lengths and make the different stages of a course clearer. It was originally designed for higher education but the principles are the same across all levels of study (see Figure 2). Students need to progress through each stage which has general rules of engagement. The definitions of each stage and the mapping of each stage by length of course are in the accompanying document in Appendix 2.
Figure 2: The Student Experience Transitions Model. Source: Morgan 2012
The benefits for students are consistency and understanding what is expected for their course. At each key transition stage, students would understand what is expected by reflecting on what they have previously learnt, how the coming year builds on what they already know and what they will achieve at the end.
Taking the opportunity to integrate
The Curriculum Review provides a real opportunity to join up each level of study and provide clarity for all stakeholders. Importantly, a National Learning Framework could provide and help with the Government’s aims of balancing ambition, relevance, flexibility and inclusivity for all learners regardless of level of study.
By Dr Adrian Gonzalez (@AGonzalez05) Senior Lecturer in Sustainability and Director of Learning and Teaching, Department of Environment and Geography at the University of York.
Climate hypocrisy in Higher Education
The climate crisis and global attempts at strengthening the sustainable and low-carbon transition is arguably the most critical issue we face and there is clear evidence to show strong Higher Education (HE) support for this twin approach. However, HE, particularly in the Global North, faces increasing scrutiny and critique over its implementation of the sustainability agenda. This has led to accusations of greenwashing, in which universities (willingly or perhaps erroneously) overmarket and/or underdeliver their sustainability policies, and climate hypocrisy, where an internationalist agenda frames student recruitment (the drive towards overseas markets), research activities and partnerships. For example, in UK tertiary education (further education and higher education), the largest sources of travel emissions are student flights, but there has been limited focus on the emissions stemming from learning and teaching, particularly fieldtrips, which this post is keen to reflect on.
Destination long haul; Higher Education residential undergraduate student fieldtrips
Outdoor education, particularly fieldtrips, offer a wide array of learner benefits and can be integral to different undergraduate programmes such as Geography, Earth and Environmental Sciences (GEES), archaeology, history and classics. However, the competitive UK higher education market has helped generate an internationalisation of undergraduate fieldtrips which are now used as a critical marketing tool to attract prospective students, who as ‘consumers’, are increasingly keen on knowing where these trips go to inform applications. For example, a brief internet search of UK GEES departments shows undergraduate trips heading to exotic locations such as the Amazonia region, Colombia (BSc Environmental Science), Bahamas (BSc Ocean Science and Marine Conservation) and Malawi (BA Human Geography).
Climate hypocrisy is evident here; students are studying programmes that acknowledge and grapple with the climate crisis and the need for transformational structural changes, yet at the same time will be enrolled on degrees that facilitate long-haul international learning opportunities without significant acknowledgement or reflection of the environmental impacts. Whilst there is no reliable publicly available data on the level of carbon emissions generated by GEES and other subject fieldtrips in UK higher education, I can give an indication by drawing on a case study of the department I work in.
Department of Environment and Geography, University of York
The department runs a wide variety of one-day and residential fieldtrips across its undergraduate and postgraduate programmes. It is the undergraduate residential trips that, owing to their design, have particularly significant carbon emissions and were made the focus of the subsequent investigation. Until 2022-2023, the department ran several residential fieldtrips that encompassed both UK and overseas destinations for its four undergraduate programmes (BSc Environmental Science; BSc Physical Geography and Environment; BSc Environment, Economics and Ecology; BA Human Geography and Environment).
I used the University of York’s carbon calculator, which draws upon the UK government’s Department for Environment, Food and Rural Affairs greenhouse conversion factors to calculate the carbon emissions stemming from travel and accommodation and the offsetting requirements. The table below shows the residential fieldtrips and carbon emissions from travel (including coach and flights where relevant) and accommodation on a per-person and 50-person basis. For four 50-person trips, this generated 108,521.85 kg CO2e (or 109 metric tonnes rounded up), equating to a carbon offsetting cost of £3,437.97 for the Department on an annual basis.
Table 1: Department of Environment and Geography, University of York fieldtrips up to 2022-2023
What does this total figure equate to? A good comparison is the Stockholm Environment Institute (SEI), an international non-profit that focuses on environment and development challenges and employs 170 staff working across several international regional centres. At the time of these fieldtrips operating, SEI’s 2020 annual report indicated that its air travel emissions were almost 550 metric tonnes CO2e (in 2019). So these department fieldtrips made up the equivalent of almost 20% of the total air travel emissions of a major international research organisation.
Conclusion: a call to action
These figures indicate the scale of the socio-environmental impacts caused and the urgent need for UK higher education learning and teaching operations, particularly in GEES given the subject areas, to be seen as ‘walking the talk.’ There have been recent efforts to address this issue through the work of the RGS-IBG who have developed a list of voluntary principles to guide geography fieldwork, including the adoption of ‘sustainable fieldtrips’ which acknowledge the need to recognise and justify the resulting carbon impacts. Whilst it is positive to see 31 institutions signed up, this is less than half of the UK GEES departments and does not incorporate any wider disciplinary commitments.
This article raises a call to action for all learned institutions and UK HE departments operating residential fieldtrips to adopt sustainable fieldtrip principles and operations. Without system-wide change, climate hypocrisy remains unchallenged in UK higher education learning and teaching.
To support academic staff and departments, several steps towards sustainable fieldtrips can be taken:
Using this data, consider revising long-haul fieldtrip locations to relevant localised destinations that can be reached through low carbon (i.e. no flights) transport;
Publish the carbon costs on the department or university website to support wider debate and discussion of sustainable fieldtrips;
Implementing sustainable fieldtrips can lead to multiple Equity, Diversity and Inclusion (EDI) benefits, particularly around accessibility and inclusivity. Use this opportunity to review and seek to strengthen the EDI agenda.
Disseminate best practice guidance through research and conference outputs;
Lobby learned institutions to adopt sustainable fieldtrip principles that align with those adopted by the RGS-IBG;
Through these steps, UK higher education can begin to create a more holistic, robust and transparent sustainability and decarbonisation agenda.
However, these actions cannot happen in isolation or nullify wider critical discussions around the UK HE sustainability agenda. One of the most significant discussion points is the impact of international students studying in the UK, a country which is the second most popular study destination in the world. Whilst these students provide significant economic benefits to the UK economy (£41.9 billion between 2021/22) and are vital to the UK higher education business model (one in six universities get over a third of their total income from overseas students), the carbon footprint far surpasses the UK higher education fieldtrip contribution. A 2023 report from 21 UK further education and higher education providers concluded that student flights accounted for 2.2 metric tonnes of CO2e or 12% of total emissions, whilst globally, student mobility is estimated to generate at least 14 megatones of Co2e per year (14 million metric tonnes). It is clear therefore that in the UK context, there is an urgent need for a robust policy debate on UK higher education funding and student mobility, otherwise the sector’s decarbonisation agenda will remain only partially addressed through sustainable fieldtrips.
We recently wrapped up our AI Learning Design Assistant (ALDA) project. It was a marathon. Multiple universities and sponsors participated in a seven-month intensive workshop series to learn how AI can assist in learning design. The ALDA software, which we tested together as my team and I built it, was an experimental apparatus designed to help us learn various lessons about AI in education.
And learn we did. As I speak with project participants about how they want to see the work continue under ALDA’s new owner (and my new employer), 1EdTech, I’ll use this post to reflect on some lessons learned so far. I’ll finish by reflecting on possible futures for ALDA.
(If you want a deeper dive from a month before the last session, listen to Jeff Young’s podcast interview with me on EdSurge. I love talking with Jeff. Shame on me for not letting you know about this conversation sooner.)
AI is a solution that needs our problems
The most fundamental question I wanted to explore with the ALDA workshop participants was, “What would you use AI for?” The question was somewhat complicated by AI’s state when I started development work about nine months ago. Back then, ChatGPT and its competitors struggled to follow the complex directions required for serious learning design work. While I knew this shortcoming would resolve itself through AI progress—likely by the time the workshop series was completed—I had to invest some of the ALDA software development effort into scaffolding the AI to boost its instruction-following capabilities at the time. I needed something vaguely like today’s AI capabilities back then to explore the questions we were trying to answer. Such as what we could be using AI for a year from then.
Once ALDA could provide that performance boost, we came to the hard part. The human part. When we got down to the nitty-gritty of the question—What would you use this for?—many participants had to wrestle with it for a while. Even the learning designers working at big, centralized, organized shops struggled to break down their processes into smaller steps with documents the AI could help them produce. Their human-centric rules relied heavily on the humans to interpret the organizational rules as they worked organically through large chunks of design work. Faculty designing their own courses had a similar struggle. How is their work segmented? What are the pieces? Which pieces would they have an assistant work on if they had an assistant?
The answers weren’t obvious. Participants had to discover them by experimenting throughout the workshop series. ALDA was designed to make that discovery process easier.
A prompt engineering technique for educators: Chain of Inquiry
Along with the starting question, ALDA had a starting hypothesis: AI can function as a junior learning designer.
How does a junior learning designer function? It turns out that their primary tool is a basic approach that makes sense in an educator’s context and translates nicely into prompt engineering for AI.
Learning designers ask their teaching experts questions. They start with general ones. Who are your students? What is your course about? What are the learning goals? What’s your teaching style?
These questions get progressively more specific. What are the learning objectives for this lesson? How do you know when students have achieved those objectives? What are some common misconceptions they have?
Eventually, the learning designer has built a clear enough mental model that they can draft a useful design document of some form or other.
Notice the similarities and differences between this approach and scaffolding a student’s learning. Like scaffolding, Chain of Inquiry moves from the foundational to the complex. It’s not about helping the person being scaffolded with their learning, but it is intended to help them with their thinking. Specifically, the interview progression helps the educator being interviewed think more clearly about hard design problems by bringing relevant context into focus. This process of prompting the interviewee to recall salient facts relevant to thinking through challenging, detailed problems is very much like the AI prompt engineering strategy called Chain of Thought.
In the interview between the learning designer and the subject-matter expert, the chain of thought they spin together is helpful to both parties for different reasons. It helps the learning designer learn while helping the subject-matter expert recall relevant details that help with thinking. The same is true in ALDA. The AI is learning from the interview, while the same process helps both parties focus on helpful context. I call this AI interview prompt style Chain of Inquiry. I hadn’t seen it used when I first thought of ALDA and haven’t seen it used much since then, either.
In any case, it worked. Participants seem to grasp it immediately. Meanwhile, a well-crafted Chain of Inquiry prompt in ALDA produced much better documents after it elicited good information through interviews with its human partners.
Improving mental models helps
AI is often presented, sold, and designed to be used as a magic talking machine. It’s hard to imagine what you would and wouldn’t use a tool for if you don’t know what it does. We went at this problem through a combination of teaching, user interface design, and guided experimentation.
On the teaching side, I emphasized that a generative AI model is a sophisticated pattern-matching and completion machine. If you say “Knock knock” to it, it will answer “Who’s there?” because it knows what usually comes after “Knock knock.” I spent some time building up this basic idea, showing the AI matching and completing more and more sophisticated patterns. Some participants initially reacted to this lesson as “not useful” or “irrelevant.” But it paid off over time as participants experienced that understanding helped them think more clearly about what to expect from the AI, with some additional help from ALDA’s design.
ALDA’s basic structure is simple:
Prompt Templates are re-usable documents that define the Chain of Inquiry interview process (although they are generic enough to support traditional Chain of Thought as well).
Chats are where those interviews take place. This part of ALDA is similar to a typical ChatGPT-like experience, except that the AI asks questions first and provides answers later based on the instructions it receives from the Prompt Template.
Lesson Drafts are where users can save the last step of a chat, which hopefully will be the draft of some learning design artifact they want to use. These drafts can be downloaded as Word or PDF documents and worked on further by the human.
A lot of the magic of ALDA is in the prompt template page design. It breaks down the prompts into three user-editable parts:
General Instructions provide the identity of the chatbot that guides its behavior, e.g., “I am ALDA, your AI Learning Design Assistant. My role is to work with you as a thoughtful, curious junior instructional designer with extensive training in effective learning practices. Together, we will create a comprehensive first draft of curricular materials for an online lesson. I’ll assist you in refining ideas and adapting to your unique context and style.
“Important: I will maintain an internal draft throughout our collaboration. I will not display the complete draft at the end of each step unless you request it. However, I will remind you periodically that you can ask to see the full draft if you wish.
“Important Instruction: If at any point additional steps or detailed outlines are needed, I will suggest them and seek your input before proceeding. I will not deviate from the outlined steps without your approval.
“
Output template provides an outline of the document that the AI is instructed to produce at the end of the interview.
Steps provide the step-by-step process for the Chain of Inquiry.
The UI reinforces the idea of pattern matching and completion. The Output Template gives the AI the structure of the document it is trying to complete by the end of the chat. The General Instructions and Steps work together to define the interview pattern the system should imitate as it tries to complete the document.
Armed with the lesson and scaffolded by the template, participants got better over time at understanding how to think about asking the AI to do what they wanted it to do.
Using AI to improve AI
One of the biggest breakthroughs came with the release of a feature near the very end of the workshop series. It’s the “Improve” button at the bottom of the Template page.
When the user clicks on that button, it sends whatever is in the template to ChatGPT. It also sends any notes the user enters, along with some behind-the-scenes information about how ALDA templates are structured.
Template creators can start with a simple sentence or two in the General Instructions. Think of it as a starting prompt, e.g., “A learning design interview template for designing and drafting a project-based learning exercise.” The user can then tell “Improve” to create a full template based on that prompt. Because ALDA tells ChatGPT what a complete template looks like, the AI returns a full draft of all the fields ALDA needs to create a template. The user can then test that template and go back to the Improve window to ask for the AI to improve the template’s behavior or extend its functionality.
Building this cycle into the process created a massive jump in usage and creativity among the participants who used it. I started seeing more and more varied templates pop up quickly. User satisfaction also improved significantly.
So…what is it good for?
The usage patterns turned out to be very interesting. Keep in mind that this is a highly unscientific review; while I would have liked to conduct a study or even a well-designed survey, the realities of building this on the fly as a solo operator managing outsourced developers limited me to anecdata for this round.
The observations from the learning designers from large, well-orchestrated teams seem to line up with my theory that the big task will be to break down our design processes into chunks that are friendly to AI support. I don’t see a short-term scenario in which we can outsource all learning design—or replace it—with AI. (By the way, “air gapping” the AI, by which I mean conducting an experiment in which nothing the AI produced would reach students without human review, substantially reduced anxieties about AI and improved educators’ willingness to experiment and explore the boundaries.)
For the individual instructors, particularly in institutions with few or no learning designers, I was pleasantly surprised to discover how useful ALDA proved to be in the middle of the term and afterward. We tend to think about learning design as a pre-flight activity. The reality is that educators are constantly adjusting their courses on the fly and spending time at the end to tweak aspects that didn’t work the way they liked. I also noticed that educators seemed interested in using AI to make it safer for them to try newer, more challenging pedagogical experiments like project-based learning or AI-enabled teaching exercises if they had ALDA as a thought partner that could both accelerate the planning and bring in some additional expertise. I don’t know how much of this can be attributed to the pure speed of the AI-enabled template improvement loop and how much the holistic experience helped them feel they understood and had control over ALDA in a way that other tools may not offer them.
Possible futures for ALDA under 1EdTech
As for what comes next, nothing has been decided yet. I haven’t been blogging much lately because I’ve been intensely focused on helping the 1EdTech team think more holistically about the many things the organization does and many more that we could do. ALDA is a piece of that puzzle. We’re still putting the pieces in place to determine where ALDA fits in.
I’ll make a general remark about 1EdTech before exploring specific possible futures for ALDA. Historically, 1EdTech has solved problems that many of you don’t (and shouldn’t) know you could have. When your students magically appear in your LMS and you don’t have to think about how your roster got there, that was because of us. When you switch LMSs, and your students still magically appear, that’s 1EdTech. When you add one of the million billion learning applications to your LMS, that was us too. Most of those applications probably wouldn’t exist if we hadn’t made it easy for them to integrate with any LMS. In fact, the EdTech ecosystem as we know it wouldn’t exist. However much you may justifiably complain about the challenges of EdTech apps that don’t work well with each other, without 1EdTech, they mostly wouldn’t work with each other at all. A lot of EdTech apps simply wouldn’t exist for that reason.
Still. That’s not nearly enough. Getting tech out of your way is good. But it’s not good enough. We need to identify real, direct educational problems and help to make them easier and more affordable to solve. We must make it possible for educators to keep up with changing technology in a changing world. ALDA could play several roles in that work.
First, it could continue to function as a literacy teaching tool for educators. The ALDA workshops covered important aspects of understanding AI that I’ve not seen other efforts cover. We can’t know how we want AI to work in education without educators who understand and are experimenting with AI. I will be exploring with ALDA participants, 1EdTech members, and others whether there is the interest and funding we need to continue this aspect of the work. We could wrap some more structured analysis around future workshops to find out what the educators are learning and what we can learn from them.
Speaking of which, ALDA can continue to function as an experimental apparatus. Learning design is a process that is largely dark to us. It happens in interviews and word processor documents on individual hard drives. If we don’t know where people need the help—and if they don’t know either—then we’re stuck. Product developers and innovators can’t design AI-enabled products to solve problems they don’t understand.
Finally, we can learn the aspects of learning design—and teaching—that need to be taught to AI because the knowledge it needs isn’t written down in a form that’s accessible to it. As educators, we learn a lot of structure in the course of teaching that often isn’t written down and certainly isn’t formalized in most EdTech product data structures. How and when to probe for a misconception. What to do if we find one. How to give a hint or feedback if we want to get the student on track without giving away the answer. Whether you want your AI to be helping the educator or working directly with the student—which is not really an either/or question—we need AI to better understand how we teach and learn if we want it to get better at helping us with those tasks. Some of the learning design structures we need are related to deep aspects of how human brains work. Other structures evolve much more quickly, such as moving to skills-based learning. Many of these structures should be wired deep into our EdTech so you don’t have to think or worry about them. EdTech products should support them automatically. Something like ALDA could be an ongoing laboratory in which we test how educators design learning interventions, how those processes co-evolve with AI over time, and where feeding the AI evidence-based learning design structure could make it more helpful.
The first incarnation ALDA was meant to be an experiment in the entrepreneurial sense. I wanted to find out what people would find useful. It’s ready to become something else. And it’s now at a home where it can evolve. The most important question about ALDA hasn’t changed all that much:
