Category: Assessment

  • The importance of consequential feedback

    The importance of consequential feedback

    Imagine this: a business student managing a virtual company makes a poor decision, leading to a simulated bankruptcy. Across campus, a medical student adjusts a treatment in a patient simulation and observes improvements in the virtual patient’s condition.

    When students practice in a simulated real-world environment they have access to a rich set of feedback information, including consequential feedback. Consequential feedback provides vital information about the consequences of students’ actions and decisions. Typically, though, in the perennial NSS-driven hand-wringing about improving feedback in higher education, we are thinking only about evaluative feedback information – when educators or peers critique students’ work and suggest improvements.

    There’s no doubt evaluative feedback, especially corrective feedback, is important. But if we’re only talking about evaluative feedback, we are missing whole swathes of invaluable feedback information crucial to preparing graduates for professional roles.

    In a recently published, open access paper in Assessment and Evaluation in Higher Education, we make the case for educators to design for and support students in noticing, interpreting and learning from consequential feedback information.

    What’s consequential feedback?

    Consequential feedback involves noticing the connection between actions and their outcomes (consequences). For example, if we touch a hot stove, we get burned. In this example, noticing the burn is both immediate and obvious. Connecting it to the action of touching the stove is also easy – little interpretation needs to be made. However, there are many cause-effect (action-consequence) sequences embedded in professional practice that are not so easy to connect. Students may need help in noticing the linkages, interpreting them and making corrections to their actions to lead to better consequences in the future.

    For instance, the business student above might decide on a pricing strategy and observe its effect on market share. The simulation speeds up time so students can observe the effects of price change on sales and market share. In real life, observing the consequences of a pricing change might take weeks or months. Through the simulation, learners can experiment with different pricing strategies, making different assumptions about the market, and observing the effects, to build their understanding of how these two variables are linked under different conditions. Critically, they learn the importance of this linkage so they can monitor in the messier, delayed real life situations they might face as a marketing professional.

    Consequential feedback isn’t just theoretical. It is already making an impact in diverse educational fields such as healthcare, business, mathematics and the arts. But the disparate literature we reviewed almost never names this information as consequential feedback. To improve feedback in higher education, we need to be able to talk to educators and students explicitly about this rich font of feedback information. We need a language for it so we can explore how it is distinct from and complementary to evaluative feedback. Naming it allows us to deliberately practice different ways of enhancing it and build evidence about how to teach students to use it well.

    Why does it matter?

    Attending to consequential feedback shifts the focus from external judgments of quality to an internalised understanding of cause and effect. It enables students to experience the results of their decisions and use these insights to refine their practice. Thus, it forms the grist for reflective thinking and a host of twenty-first century skills needed to solve the world’s most pressing problems.

    In “real-life” after university, graduates are unlikely to have a mentor or teacher standing over them offering the kind of evaluative feedback that dominates discussion of feedback in higher education. Instead, they need to be able to learn independently from the consequential feedback readily available in the workplace and beyond. Drawing on consequential feedback information, professionals can continuously learn and adapt their practice to changing contexts. Thus, educators need to design opportunities that simulate professional practices, paying explicit attention to helping students learn from the consequential feedback afforded by these instructional designs.

    How can educators harness it?

    While consequential feedback is powerful, capitalising on it during higher education requires careful design. Here are some strategies for educators to try in their practice:

    Use simulations, role-plays, and projects: Simulations provide a controlled environment where students can explore the outcomes of their actions. For example, in a healthcare setting, students might use patient mannequins or virtual reality tools to practice diagnostic and treatment skills. In a human resources course, students might engage in mediation role plays. In an engineering course, students could design and test products like model bridges or rockets.

    Design for realism: Whenever possible, feedback opportunities should replicate real-world conditions. For instance, a law student participating in a moot court can see how their arguments hold up under cross-examination or a comedy student can see how a real audience responds to their show.

    Encourage reflection: Consequential feedback is most effective when paired with reflection. Educators can prompt students to consider questions such as: What did you do? Why? What happened when you did x? Was y what you expected or wanted? How do these results compare to professional standards? Why did you get that result? What could you change to get the results you want?

    Pair with evaluative feedback: Students may see that they didn’t get the result they wanted but not know how to correct their actions. Consequential feedback doesn’t replace evaluative feedback; it complements it. For example, after a business simulation, an instructor might provide additional guidance on interpreting KPIs or suggest strategies for improvement. This pairing helps students connect outcomes with actionable next steps.

    Shifting the frame

    Focusing on consequential feedback represents a shift in how we think about assessment, feedback, and learning itself. By designing learning experiences that allow students to act and observe the natural outcomes of their actions, we create opportunities for deeper, more meaningful engagement in the learning process. As students study the impact of their actions, they learn to take responsibility for their choices. This approach fosters the problem-solving, adaptability, independence, and professional and social responsibility they’ll need throughout their lives.

    A key question educators should be asking is: how can I help students recognise and learn from the outcomes of their actions? The answer lies in designing for and highlighting consequential feedback.

    Source link

  • Becoming a professional services researcher in HE – making the train tracks converge

    Becoming a professional services researcher in HE – making the train tracks converge

    by Charlotte Verney

    This blog builds on my presentation at the BERA ECR Conference 2024: at crossroads of becoming. It represents my personal reflections of working in UK higher education (HE) professional services roles and simultaneously gaining research experience through a Masters and Professional Doctorate in Education (EdD).

    Professional service roles within UK HE include recognised professionals from other industries (eg human resources, finance, IT) and HE-specific roles such as academic quality, research support and student administration. Unlike academic staff, professional services staff are not typically required, or expected, to undertake research, yet many do. My own experience spans roles within six universities over 18 years delivering administration and policy that supports learning, teaching and students.

    Traversing two tracks

    In 2016, at an SRHE Newer Researchers event, I was asked to identify a metaphor to reflect my experience as a practitioner researcher. I chose this image of two train tracks as I have often felt that I have been on two development tracks simultaneously –  one building professional experience and expertise, the other developing research skills and experience. These tracks ran in parallel, but never at the same pace, occasionally meeting on a shared project or assignment, and then continuing on their separate routes. I use this metaphor to share my experiences, and three phases, of becoming a professional services researcher.

    Becoming research-informed: accelerating and expanding my professional track

    The first phase was filled with opportunities; on my professional track I gained a breadth of experience, a toolkit of management and leadership skills, a portfolio of successful projects and built a strong network through professional associations (eg AHEP). After three years, I started my research track with a masters in international higher education. Studying felt separate to my day job in academic quality and policy, but the assignments gave me opportunities to bring the tracks together, using research and theory to inform my practice – for example, exploring theoretical literature underpinning approaches to assessment whilst my institution was revising its own approach to assessing resits. I felt like a research-informed professional, and this positively impacted my professional work, accelerating and expanding my experience.

    Becoming a doctoral researcher: long distance, slow speed

    The second phase was more challenging. My doctoral journey was long, taking 9 years with two breaks. Like many part-time doctoral students, I struggled with balance and support, with unexpected personal and professional pressures, and I found it unsettling to simultaneously be an expert in my professional context yet a novice in research. I feared failure, and damaging my professional credibility as I found my voice in a research space.

    What kept me going, balancing the two tracks, was building my own research support network and my researcher identity. Some of the ways I did this was through zoom calls with EdD peers for moral support, joining the Society for Research into Higher Education to find my place in the research field, and joining the editorial team of a practitioner journal to build my confidence in academic writing.

    Becoming a professional services researcher: making the tracks converge

    Having completed my doctorate in 2022, I’m now actively trying to bring my professional and research tracks together. Without a roadmap, I’ve started in my comfort-zone, sharing my doctoral research in ‘safe’ policy and practitioner spaces, where I thought my findings could have the biggest impact. I collaborated with EdD peers to tackle the daunting task of publishing my first article. I’ve drawn on my existing professional networks (ARC, JISC, QAA) to establish new research initiatives related to my current practice in managing assessment. I’ve made connections with fellow professional services researchers along my journey, and have established an online network  to bring us together.

    Key takeaways for professional services researchers

    Bringing my professional experience and research tracks together has not been without challenges, but I am really positive about my journey so far, and for the potential impact professional services researchers could have on policy and practice in higher education. If you are on your own journey of becoming a professional services researcher, my advice is:

    • Make time for activities that build your research identity
    • Find collaborators and a community
    • Use your professional experience and networks
    • It’s challenging, but rewarding, so keep going!

    Charlotte Verney is Head of Assessment at the University of Bristol. Charlotte is an early career researcher in higher education research and a leader in within higher education professional services. Her primary research interests are in the changing nature of administrative work within universities, using research approaches to solve professional problems in higher education management, and using creative and collaborative approaches to research. Charlotte advocates for making the academic research space more inclusive for early career and professional services researchers. She is co-convenor of the SRHE Newer Researchers Network and has established an online network for higher education professional services staff engaged with research.

    Author: SRHE News Blog

    An international learned society, concerned with supporting research and researchers into Higher Education

    Source link

  • Alternatives to the essay can be inclusive and authentic

    Alternatives to the essay can be inclusive and authentic

    I lead our largest optional final-year module – Crime, Justice and the Sex Industry – with 218 registered students for the 24–25 academic year.

    That is a lot of students to assess.

    For that module in the context, I was looking for an assessment that is inclusive, authentic, and hopefully enjoyable to write.

    I also wanted to help make students into confident writers, who make writing a regular practice with ongoing revisions.

    Inspired by the wonderful Katie Tonkiss at Aston University, I devised a letter assessment for our students.

    This was based on many different pedagogical considerations, and the acute need to teach students how to hold competing and conflicting harms and needs in tension. I consider the sex industry within the broader context of violence against women and girls.

    The sex industry, sexual exploitation, and violence against women in girls are brutal and traumatic topics that can incite divisive responses.

    Now more than ever, we need to be able to deal well with differences, to negotiate, to encourage, to reflect, and to try and move discussions forward, as opposed to shutting them down.

    Their direction and pace

    During the pandemic, I designed my module based on a non-linear pedagogy – giving students the power to navigate teaching resources at a direction and pace of their choosing.

    This has strong EDI principles, and was strongly led by my own dyslexia. I recognised that students often have constraints on their time and energy levels that mean they need to engage with learning in different patterns during different weeks – disclosing that they “binge watch” lecture videos, podcasts, or focus heavily on texts during certain weeks to block out their time.

    The approach also honours the principles of trauma-informed teaching, empowering students to navigate sensitive topics of gender-based and sexual violence.

    As I argue here with Lisa Anderson, students are now learning in a post-pandemic context with differing expectations, accessibility needs, and barriers including paid work responsibilities.

    The “full-time student” is now something of an anachronism, and education must meet this new reality – there are now more students in paid employment than not according to the 2023 HEPI/Advance HE Student Academic Experience Study.

    We have to meet students where they are, and, presently, that is in a difficult place as many students struggle with the cost of living and the battle to “have it all”.

    Students may not be asked to write exam answers or essays in their post-university life, but they will certainly be expected to write persuasively, convincingly, engaging with multiple viewpoints, and sitting with their own discomfort.

    This may take the form of webpage outputs, summaries, policy briefings, strategy documents, emails to stakeholders, campaign materials. As such, students are strongly encouraged to think about the letter from day one of semester, and consider who their recipient will be.

    They are told that it is easier to write such a letter to somebody with an opposing viewpoint – laying out your case in a respectful, warm and supportive way to try and progress the discussion. Students are also encouraged to acknowledge their own positionality, and share this if desirable, including if they can identify a thinker, document or moment that changed their position.

    Working towards change

    An example is a student who holds a position influenced by their faith, writing their letter to a faith leader or family member, acknowledging that they respect their beliefs, but strongly endorsing an approach that places harm-reduction and safety first. Finding a place of agreement and building from there, and accepting that working towards change can be a long process.

    Another example is a student who holds sex industry abolitionist views, writing to a sex worker, expressing concern and solidarity with the multiple forms of harm, stigma and violence they have experienced, including institutional violence.

    They consider how the law itself facilitates the context that makes violence more likely to occur. This is particularly pertinent at the moment as we experience a fresh wave of digital “me too” and high-profile cases of sexual violence and victim-blaming.

    In this way, students are taught to examine different documents and evidence, from legal, policy, charity briefings and statements, journal articles, books, reports, documentaries, global sex worker grassroots initiatives, news reports, social media campaigns and footage, art, literature, etc.

    By engaging with different types of sources, we challenge the idea that academic material is top of the knowledge hierarchy, and platform the voices who often go unheard, including sex workers globally.

    The students cross-reference resources, and identify forms of harm, violence and discrimination that may not make official narratives. This also encourages students to be active members of our community, contributing to each workshop either verbally or digitally, in real-time, or asynchronously via our class-wide google doc.

    Students are also taught that it is OK to not have the definitive answer, and to instead ask the recipient to help them further their knowledge. They are also taught that it is ok to change our position and recommendations depending on what evidence we encounter.

    Above all, they are taught that two things can be true at the same time: something might be harmful, and the response to it awful too.

    Students responded overwhelmingly in favour of this approach, and many expressed a new-found love of writing, and reading. Engaging with many different mediums including podcasts, tweets, reels, history talks, art exhibitions, gave them confidence in their reading and study skills.

    Putting choice and enjoyment in the curriculum is not about “losing academic rigour”, it is about firing students up for their topics of study, and ensuring they can communicate powerfully to different audiences using different tools.

    Dear me, I wish we had tried this assessment sooner. xoxo

    Source link

  • Gaps in sustainability literacy in non-STEM higher education programmes

    Gaps in sustainability literacy in non-STEM higher education programmes

    by Erika Kalocsányiová and Rania Hassan

    Promoting sustainability literacy in higher education is crucial for deepening students’ pro-environmental behaviour and mindset (Buckler & Creech, 2014; UNESCO, 1997), while also fostering social transformation by embedding sustainability at the core of the student experience. In 2022, our group received an SRHE Scoping Award to synthesise the literature on the development, teaching, and assessment of sustainability literacy in non-STEM higher education programmes. We conducted a multilingual systematic review of post-2010 publications from the European Higher Education Area (EHEA), with the results summarised in Kalocsányiová et al (2024).

    Out of 6,161 articles that we identified as potentially relevant, 92 studies met the inclusion criteria and are reviewed in the report. These studies involved a total of 11,790 participants and assessed 9,992 university programmes and courses. Our results suggest a significant growth in research interest in sustainability in non-STEM fields since 2017, with 75 studies published compared to just 17 in the preceding seven years. Our analysis also showed that Spain, the United Kingdom, Germany, Turkey, and Austria had the highest concentration of publications, with 25 EHEA countries represented in total. The 92 reviewed studies were characterised by high methodological diversity: nearly half employed quantitative methods (47%), followed by qualitative studies (40%) and mixed methods research (13%). Curriculum assessments using quantitative content analysis of degree and course descriptors were among the most common study types, followed by surveys and intervention or pilot studies. Curriculum assessments provided a systematic way to evaluate the presence or absence of sustainability concepts within curricula at both single HE institutions and in comparative frameworks. However, they often captured only surface-level indications of sustainability integration into undergraduate and postgraduate programmes, without providing evidence on actual implementation and/or the effectiveness of different initiatives. Qualitative methods, including descriptive case studies and interviews that focused on barriers, challenges, implementation strategies, and the acceptability of new sustainability literacy initiatives, made up 40% of the current research. Mixed methods studies accounted for 13% of the reviewed articles, often applying multiple assessment tools simultaneously, including quantitative sustainability competency assessment instruments combined with open-ended interviews or learning journals.

    In terms of disciplines, Economics, Business, and Administrative Studies held the largest share of reviewed studies (26%), followed by Education (23%). Multiple disciplines accounted for 22% of the reviewed publications, reflecting the interconnected nature of sustainability. Finance and Accounting contributed only 6%, indicating a need for further research. Similarly, Language and Linguistics, Mass Communication and Documentation, and Social Sciences collectively represented only 12% of the reviewed studies. Creative Arts and Design with just 2% was also a niche area. Although caution should be exercised when drawing conclusions from these results, they highlight the need for more research within the underrepresented disciplines. This in turn can help promote awareness among non-STEM students, stimulate ethical discussions on the cultural dimensions of sustainability, and encourage creative solutions through interdisciplinary dialogue.

    Regarding factors and themes explored, the studies focused primarily on the acquisition of sustainability knowledge and competencies (27%), curriculum assessment (23%), challenges and barriers to sustainability integration (10%), implementation and evaluation research (10%), changes in students’ mindset (9%), key competences in sustainability literacy (5%), and active student participation in Education for Sustainable Development (5%). In terms of studies discussing acquisition processes, key focus areas included the teaching of Sustainable Development Goals, awareness of macro-sustainability trends, and knowledge of local sustainability issues. Studies on sustainability competencies focussed on systems thinking, critical thinking, problem-solving skills, ethical awareness, interdisciplinary knowledge, global awareness and citizenship, communication skills, and action-oriented mindset. These competencies and knowledge, which are generally considered crucial for addressing the multifaceted challenges of sustainability (Wiek et al., 2011), were often introduced to non-STEM students through stand-alone lectures, workshops, or pilot studies involving new cross-disciplinary curricula.

    Our review also highlighted a broad range of pedagogical approaches adopted for sustainability teaching and learning within non-STEM disciplines. These covered case and project-based learning, experiential learning methods, problem-based learning, collaborative learning, reflection groups, pedagogical dialogue, flipped classroom approaches, game-based learning, and service learning. While there is strong research interest in the documentation and implementation of these pedagogical approaches, few studies have so far attempted to assess learning outcomes, particularly regarding discipline-specific sustainability expertise and real-world problem-solving skills.

    Many of the reviewed studies relied on single-method approaches, meaning valuable insights into sustainability-focused teaching and learning may have been missed. For instance, studies often failed to capture the complexities surrounding sustainability integration into non-STEM programs, either by presenting positivist results that require further contextualisation or by offering rich context limited to a single course or study group, which cannot be generalised. The assessment tools currently used also seemed to lack consistency, making it difficult to compare outcomes across programmes and institutions to promote best practices. More robust evaluation designs, such as longitudinal studies, controlled intervention studies, and mixed methods approaches (Gopalan et al, 2020; Ponce & Pagán-Maldonado, 2015), are needed to explore and demonstrate the pedagogical effectiveness of various sustainability literacy initiatives in non-STEM disciplines and their impact on student outcomes and societal change.

    In summary, our review suggests good progress in integrating sustainability knowledge and competencies into some core non-STEM disciplines, while also highlighting gaps. Based on the results we have formulated some questions that may help steer future research:

    • Are there systemic barriers hindering the integration of sustainability themes, challenges and competencies into specific non-STEM fields?
    • Are certain disciplines receiving disproportionate research attention at the expense of others?
    • How do different pedagogical approaches compare in terms of effectiveness for fostering sustainability literacy in and across HE fields?
    • What new educational practices are emerging, and how can we fairly assess them and evidence their benefits for students and the environment?

    We also would like to encourage other researchers to engage with knowledge produced in a variety of languages and educational contexts. The multilingual search and screening strategy implemented in our review enabled us to identify and retrieve evidence from 25 EHEA countries and 24 non-English publications. If reviews of education research remain monolingual (English-only), important findings and insights will go unnoticed hindering knowledge exchange, creativity, and innovation in HE.

    Dr. Erika Kalocsányiová is a Senior Research Fellow with the Institute for Lifecourse Development at the University of Greenwich, with research centering on public health and sustainability communication, migration and multilingualism, refugee integration, and the implications of these areas for higher education policies.

    Rania Hassan is a PhD student and a research assistant at the University of Greenwich. Her research centres on exploring enterprise development activities within emerging economies. As a multidisciplinary and interdisciplinary researcher, Rania is passionate about advancing academia and promoting knowledge exchange in higher education.

    Author: SRHE News Blog

    An international learned society, concerned with supporting research and researchers into Higher Education

    Source link

  • the future of learning design. – Sijen

    the future of learning design. – Sijen

    There is a looming skills deficit across all disciplines currently being taught in Universities today. The vast majority of degree programmes are, at best, gradual evolutions of what has gone before. At their worst they are static bodies of knowledge transmission awaiting a young vibrant new member of faculty to reignite them. Internal reviews are too often perfunctory exercises, seldom challenging the future direction of graduates as long as pass rates are sustained. That is until is to late and failure rates point to a ‘problem’ at a fundamental level around a degree design.

    We, collectively, are at the dawn of a new knowledge-skills-cognition revolution. The future of the professionals has been discussed for some years now. It will be a creeping, quiet, revolution (Susskind and Susskind, 2017). Although we occasionally hear about some fast food business firing all of its front-of-house staff in favour of robotic manufacturing processes and A.I. Ordering services, the reality is that in the majority of contexts the intelligent deployment of A.I. to enhance business operations requires humans to describe how these systems operate with other humans. This is because at present none of these systems score highly on any markers or Emotional Intelligence or EQ.

    Image generaed by Windows Copilot

    Arguably it has become increasingly important to ensure that graduates from any and all disciplines have been educated as to how to describe what they do and why they do it. They need to develop a higher degree of comfort with articulating each thought process and action taken. To do this we desperately need course and programme designers to desist from just describing (and therefore assessing) purely cognitive (intellectual) skills as described by Bloom et.al, and limit themselves to one or two learning outcomes using those formulations. Instead they need to elevate the psychomotor skills in particular, alongside an increasing emphasis on interpersonal ones.

    Anyone who has experimented with prompting any large language model (LLM) will tell you the language used falls squarely under the psychomotor domain. At the lowest levels one might ask to match, copy, imitate, then at mid-levels of skill deployment one might prompt a system to organise, calibrate, compete or show, rating to the highest psychomotor order of skills to ask A.I. systems to define, specify, even imagine. This progressive a type of any taxonomy allows for appropriate calibration of input and output. The ability to use language, to articulate, is an essential skill. There are some instructive (ad entertaining) YouTube videos of parents supporting their children to write instructions (here’s a great example), a skill that is seldom further developed as young people progress into tertiary studies.

    Being able to assess this skill is also challenging. When one was assessing text-based comprehension, even textual analysis, then one could get away with setting an essay question and having a semi-automated process for marking against a rudimentary rubric. Writing instructions, or explanations, of the task carried out, is not the same as verbally describing the same task. Do we imagine that speech recognition technology won’t become an increasingly part of many productive job roles. Not only do courses and programmes need to be designed around a broader range of outcomes, we also need to be continuously revising our assessment opportunities for those outcomes.

    References

    Susskind, R., & Susskind, D. (2017). The Future of the Professions: How Technology Will Transform the Work of Human Experts (Reprint edition). OUP Oxford.

    Source link