Tag: Assessment

  • Cheating matters but redrawing assessment “matters most”

    Cheating matters but redrawing assessment “matters most”

    Conversations over students using artificial intelligence to cheat on their exams are masking wider discussions about how to improve assessment, a leading professor has argued.

    Phillip Dawson, co-director of the Centre for Research in Assessment and Digital Learning at Deakin University in Australia, argued that “validity matters more than cheating,” adding that “cheating and AI have really taken over the assessment debate.”

    Speaking at the conference of the U.K.’s Quality Assurance Agency, he said, “Cheating and all that matters. But assessing what we mean to assess is the thing that matters the most. That’s really what validity is … We need to address it, but cheating is not necessarily the most useful frame.”

    Dawson was speaking shortly after the publication of a survey conducted by the Higher Education Policy Institute, which found that 88 percent of U.K. undergraduates said they had used AI tools in some form when completing assessments.

    But the HEPI report argued that universities should “adopt a nuanced policy which reflects the fact that student use of AI is inevitable,” recognizing that chat bots and other tools “can genuinely aid learning and productivity.”

    Dawson agreed, arguing that “assessment needs to change … in a world where AI can do the things that we used to assess,” he said.

    Referencing—citing sources—may be a good example of something that can be offloaded to AI, he said. “I don’t know how to do referencing by hand, and I don’t care … We need to take that same sort of lens to what we do now and really be honest with ourselves: What’s busywork? Can we allow students to use AI for their busywork to do the cognitive offloading? Let’s not allow them to do it for what’s intrinsic, though.”

    It was a “fantasy land” to introduce what he called “discursive” measures to limit AI use, where lecturers give instructions on how AI use may or may not be permitted. Instead, he argued that “structural changes” were needed for assessments.

    “Discursive changes are not the way to go. You can’t address this problem of AI purely through talk. You need action. You need structural changes to assessment [and not just a] traffic light system that tells students, ‘This is an orange task, so you can use AI to edit but not to write.”

    “We have no way of stopping people from using AI if we aren’t in some way supervising them; we need to accept that. We can’t pretend some sort of guidance to students is going to be effective at securing assessments. Because if you aren’t supervising, you can’t be sure how AI was or wasn’t used.”

    He said there are three potential outcomes for the impact on grades as AI develops: grade inflation, where people are going to be able to do “so much more against our current standards, so things are just going to grow and grow”; and norm referencing, where students are graded on how they perform compared to other students.

    The final option, which he said was preferable, was “standards inflation,” “where we just have to keep raising the standards over time, because what AI plus a student can do gets better and better.”

    Over all, the impact of AI on assessments is fundamental, he said, adding, “The times of assessing what people know are gone.”

    Source link

  • Achieving a 100% Completion Rate for Student Assessment at the University of Charleston

    Achieving a 100% Completion Rate for Student Assessment at the University of Charleston

    Seated in beautiful Charleston, West Virginia, the University of Charleston (UC) boasts “a unique opportunity for those who want an exceptional education in a smaller, private setting.” UC provides a unique student experience focused on retention and student success even before students arrive on campus.

    Students are offered an opportunity to complete the College Student Inventory (CSI) online through a pre-orientation module. This initiative is reinforced through the student’s Success and Motivation first-year course. University instructors serve as mentors, utilizing the CSI results to capitalize on insights related to each individual student’s strengths and opportunities for success through individual review meetings and strategic support and skill building structured within this course.

    After achieving a 7% increase in retention, Director of Student Success and First-Year Programs Debbie Bannister says administering the CSI each year is non-negotiable. Additionally, the campus has refocused on retention, emphasizing, “Everyone has to realize that they are part of retention, and they’re part of keeping every single student on our campus.”

    UC has reinstated a Retention Committee that utilizes summary information from the CSI to understand the needs of its students. Of particular concern, UC notes that the transfer portal has created additional challenges with upperclassmen, so including a representative from the athletic department on the retention committee has been crucial.

    Through this focus on retention and strong implementation strategy, UC achieves a 100% completion rate for the CSI for their first-year student cohort. Building off the scaffolding support from early support meetings related to the CSI insights, first-year instructors are able to refer back to reinforce articulated support strategies and goals throughout the first-year experience. The structure and progression through this course reiterates college preparation skills and resources building motivation and a growth mindset to persist through college.

    Increase student success through early intervention

    Join institutions such as the University of Charleston by using the College Student Inventory with your incoming students. More than 1,400 institutions have used the CSI, and it’s been taken by more than 2.6 million students nationwide. Learn more about how you can use it to intervene earlier with students and increase student yield.

    Source link

  • Another way of thinking about the national assessment of people, culture, and environment

    Another way of thinking about the national assessment of people, culture, and environment

    There is a multi-directional relationship between research culture and research assessment.

    Poor research assessment can lead to poor research cultures. The Wellcome Trust survey in 2020 made this very clear.

    Assessing the wrong things (such as a narrow focus on publication indicators), or the right things in the wrong way (such as societal impact rankings based on bibliometrics) is having a catalogue of negative effects on the scholarly enterprise.

    Assessing the assessment

    In a similar way, too much research assessment can also lead to poor research cultures. Researchers are one of the most heavily assessed professions in the world. They are assessed for promotion, recruitment, probation, appraisal, tenure, grant proposals, fellowships, and output peer review. Their lives and work are constantly under scrutiny, creating competitive and high-stress environments.

    But there is also a logic (Campbell’s Law) that tells us that if we assess research culture it can lead to greater investment into improving it. And it is this logic that the UK Joint HE funding bodies have drawn on in their drive to increase the weighting given to the assessment of People, Culture & Environment in REF 2029. This makes perfect sense: given the evidence that positive and healthy research cultures are a thriving element of Research Excellence, it would be remiss of any Research Excellence Framework not to attempt to assess, and therefore incentivise them.

    The challenge we have comes back to my first two points. Even assessing the right things, but in the wrong way, can be counterproductive, as may increasing the volume of assessment. Given research culture is such a multi-faceted concept, the worry is that the assessment job will become so huge that it quickly becomes burdensome, thus having a negative impact on those research cultures we want to improve.

    It ain’t what you do, it’s the way that you do it

    Just as research culture is not so much about the research that you do but the way that you do it, so research culture assessment should concern itself not so much with the outcomes of that assessment but with the way the assessment takes place.

    This is really important to get right.

    I’ve argued before that research culture is a hygiene factor. Most dimensions of culture relate to standards that it’s critically important we all get right: enabling open research, dealing with misconduct, building community, supporting collaboration, and giving researchers the time to actually do research. These aren’t things for which we should offer gold stars but basic thresholds we all should meet. And to my mind they should be assessed as such.

    Indeed this is exactly how the REF assessed open research in 2021 (and will do so again in 2029). They set an expectation that 95 per cent of qualifying outputs should be open access, and if you failed to hit the threshold, excess closed outputs were simply unclassified. End of. There were no GPAs for open access.

    In the tender for the PCE indicator project, the nature of research culture as a hygiene factor was recognised by proposing “barrier to entry” measures. The expectation seemed to be that for some research culture elements institutions would be expected to meet a certain threshold, and if they failed they would be ineligible to even submit to REF.

    Better use of codes of practice

    This proposal did not make it into the current PCE assessment pilot. However, the REF already has a “barrier to entry” mechanism, of course, which is the completion of an acceptable REF Code of Practice (CoP).

    An institution’s REF CoP is about how they propose to deliver their REF, not how they deliver their research (although there are obvious crossovers). And REF have distinguished between the two in their latest CoP Policy module governing the writing of these codes.

    But given that REF Codes of Practice are now supposed to be ongoing, living documents, I don’t see why they shouldn’t take the form of more research-focussed (rather than REF-focussed) codes. It certainly wouldn’t harm research culture if all research performing organisations had a thorough research code of practice (most do of course) and one that covers a uniform range of topics that we all agree are critical to good research culture. This could be a step beyond the current Terms & Conditions associated with QR funding in England. And it would be a means of incentivising positive research cultures without ‘grading’ them. With your REF CoP, it’s pass or fail. And if you don’t pass first time, you get another attempt.

    Enhanced use of culture and environment data

    The other way of assessing culture to incentivise behaviours without it leading to any particular rating or ranking is to simply start collecting & surfacing data on things we care about. For example, the requirement to share gender pay gap data and to report misconduct cases, has focussed institutional minds on those things without there being any associated assessment mechanism. If you check out the Higher Education Statistics Agency (HESA) data on proportion of male:female professors, in most UK institutions you can see the ratio heading in the right direction year on year. This is the power of sharing data, even when there’s no gold or glory on offer for doing so.

    And of course, the REF already has a mechanism to share data to inform, but not directly make an assessment, in the form of ’Environment Data’. In REF 2021, Section 4 of an institution’s submission was essentially completed for them by the REF team by extracting from the HESA data, the number of doctoral degrees awarded (4a) and the volume of research income (4b); and from the Research Councils, the volume of research income in kind (4c).

    This data was provided to add context to environment assessments, but not to replace them. And it would seem entirely sensible to me that we identify a range of additional data – such as the gender & ethnicity of research-performing staff groups at various grades – to better contextualise the assessment of PCE, and to get matters other than the volume of research funding up the agendas of senior university committees.

    Context-sensitive research culture assessment

    That is not to say that Codes of Practice and data sharing should be the only means of incentivising research culture of course. Culture was a significant element of REF Environment statements in 2021, and we shouldn’t row back on it now. Indeed, given that healthy research cultures are an integral part of research excellence, it would be remiss not to allocate some credit to those who do this well.

    Of course there are significant challenges to making such assessments robust and fair in the current climate. The first of these is the complex nature of research culture – and the fact that no framework is going to cover every aspect that might matter to individual institutions. Placing boundaries around what counts as research culture could mean institutions cease working on agendas that are important to them, because they ostensibly don’t matter to REF.

    The second challenge is the severe and uncertain financial constraints currently faced by the majority of UK HEIs. Making the case for a happy and collaborative workforce when half are facing redundancy is a tough ask. A related issue here is the hugely varying levels of research (culture) capital across the sector as I’ve argued before. Those in receipt of a £1 million ‘Enhancing Research Culture’ fund from Research England, are likely to make a much better showing than those doing research culture on a shoe-string.

    The third is that we are already half-way through this assessment period and we’re only expected to get the final guidance in 2026 – two years prior to submission. And given the financial challenges outlined above, this is going to make this new element of our submission especially difficult. It was partly for this reason that some early work to consider the assessment of research culture was clear that this should celebrate the ‘journey travelled’, rather than a ‘destination achieved’.

    For this reason, to my mind, the only thing we can reasonably expect all HEIs to do right now with regards to research culture is to:

    • Identify the strengths and challenges inherent within your existing research culture;
    • Develop a strategy and action plan(s) by which to celebrate those strengths and address those challenges;
    • Agree a set of measures by which to monitor your progress against your research culture ambitions. These could be inspired by some of the suggestions resulting from the Vitae & Technopolis PCE workshops & Pilot exercise;
    • Describe your progress against those ambitions and measures. This could be demonstrated both qualitatively and quantitatively, through data and narratives.

    Once again, there is an existing REF assessment mechanism open to us here, and that is the use of the case study. We assess research impact by effectively asking HEIs to tell us their best stories – I don’t see why we shouldn’t make the same ask of PCE, at least for this REF.

    Stepping stone REF

    The UK joint funding bodies have made a bold and sector-leading move to focus research performing organisations’ attention on the people and cultures that make for world-leading research endeavours through the mechanism of assessment. Given the challenges we face as a society, ensuring we attract, train, and retain high quality research talent is critical to our success. However, the assessment of research culture has the power both to make things better or worse: to incentivise positive research cultures or to increase burdensome and competitive cultures that don’t tackle all the issues that really matter to institutions.

    To my mind, given the broad range of topics that are being worked on by institutions in the name of improving research culture, and where we are in the REF cycle, and the financial constraints facing the sector, we might benefit from a shift in the mechanisms proposed to assess research culture in 2029 and to see this as a stepping stone REF.

    Making better use of existing mechanisms such as a Codes of Practice and Environment and Culture data would assess the “hygiene factor” elements of culture without unhelpfully associating any star ratings to them. Ratings should be better applied to the efforts taken by institutions to understand, plan, monitor, and demonstrate progress against their own, mission-driven research culture ambitions. This is where the real work is and where real differentiations between institutions can be made, when contextually assessed. Then, in 2036, when we can hope that the sector will be in a financially more stable place, and with ten years of research culture improvement time behind us, we can assess institutions against their own ambitions, as to whether they are starting to move the dial on this important work.

    Source link

  • Direct and Indirect Assessment Measures of Student Learning in Higher Education – Faculty Focus

    Direct and Indirect Assessment Measures of Student Learning in Higher Education – Faculty Focus

    Source link

  • How to Implement Diagnostic Assessment Examples

    How to Implement Diagnostic Assessment Examples

    Diagnostic assessment examples can help ground the concepts of diagnostic assessment in education. Plus, these examples can be an effective tool in gauging student progress and comprehension of course concepts. 

    What is Diagnostic Assessment in Education?

    An essential part of course planning for instructors to consider is how to gauge student understanding of course concepts. At the beginning of each academic term, it’s important to consider the upcoming curriculum, and how to best assess students.

    Diagnostic assessment typically takes place at the start of a semester to evaluate a student’s current level of knowledge and skills and their strengths and weaknesses on a particular topic.

    Similar to ipsative assessments, where professors examine students’ prior work in order to assess their current knowledge and abilities, diagnostic assessments are a type of “assessment as learning.” This is distinct from “assessments of learning” or “assessments for learning.”

    Distinction Between Different Types of Assessments

    Assessment for learning, also known as formative assessments, make use of information about student progress to improve and support student learning and guide instructional strategies. They are generally instructor-driven but are for student and instructor use. Assessments for learning can occur throughout the teaching and learning process, using a variety of platforms and tools. They engage instructors in providing differentiated instruction and provide feedback to students to enhance their learning.

    Assessment as learning (formative assessment) involves active student reflection on learning, monitoring of their own progress by supporting students to critically analyze and evaluate their own learning. Contrarily, they are student-driven and occur throughout the learning process. 

    Assessment of learning (summative assessment) involves evidence of student learning to make judgments about student progress. They provide instructors with the opportunity to report evidence of meeting course objectives and typically occur at the end of a learning cycle using a variety of tools. The evaluation compares assessment information against criteria based on curriculum outcomes for the purpose of communicating to students about student progress and making informed decisions about the teaching and learning process.

    What are Diagnostic Assessments Used For?

    Students may write examples of diagnostic assessments to help professors gain insight into their existing awareness and capabilities both preceding and following instruction. As such, a diagnostic evaluation can be either:

    • A pre-course diagnostic assessment
    • A post-course diagnostic assessment

    Upon completion of a post-course diagnostic assessment, a professor can compare it against the student’s pre-course diagnostic assessment for that same course and semester in order to identify possible improvements in various specific areas. Professors can then use this information to adjust and adapt their curricula to better meet the needs of future students.

    Professors can utilize diagnostic assessment in education to plan individualized learning experiences for each student that provide both efficient and meaningful instruction.

    Examples of Diagnostic Assessment Tools and Technologies

    There are many different educational tools and technologies that enable professors and students to get instant results from learning, including Top Hat, Socrative, Kahoot, Quizziz, Mentimeter and Quizlet. Within each of these tools and technologies are several different examples of diagnostic assessments you can apply to various disciplines.

    Diagnostic Assessment Examples

    Diagnostic assessments can be conducted in many different ways, including as sets of written questions, such as in short answer or multiple choice form, as well as reflection exercises, long answer questions and creative projects.

    In courses containing group work, useful types of diagnostic assessments may include self-assessments in which group members each rate themselves based on various guidelines. The group then collects specific samples of each member’s prior work to understand the member’s mindset that led that member to give him or herself that rating.

    Different types of diagnostic assessments include:

    • Anticipation guides
    • Conference/interview
    • Formal assessment
    • Gap-closing
    • Graffiti walls
    • Journals
    • KWL
    • Mind maps
    • Parallel activity
    • Performance tasks
    • Posters
    • Quiz/test
    • Student surveys
    • Word splash

    Below, we share examples of how diagnostic assessments can be implemented in different disciplines, as well as easy-to-use tools that streamline the assessment design process for instructors.

    Diagnostic Assessment Examples for Physics

    In physics courses, instructors issue a set of conceptual questions to students at the start of the semester in order to assess the students’ current understanding of the fundamentals of physics.

    In certain educational disciplines, standardized diagnostic assessment examples have been developed that instructors can use for any course within that discipline. In physics, one of the most commonly used examples of diagnostic assessment is the Force Concept Inventory, which contains question sets about concepts, like gravity, velocity, mass and force, which are typically taught in a basic first-semester Newtonian physics course.

    Tools for Diagnostic Assessments in Physics

    Physics instructors can use Top Hat’s Polls and Quizzes feature to design diagnostic evaluations that engage students effectively. Use polls to demonstrate student understanding and see which course concepts may need further review. Frequent quizzes can be used to help students challenge themselves.

    Top Hat’s surveys and polls tools include checkpoints to help break lectures up into more manageable chunks, prompt discussions and motivate students to apply what they learn. Top Hat’s in-class polls and quizzes are multimedia-rich, helping professors engage students fully in the learning and assessment process. Examples of diagnostic assessment in education using these tools include click-on-target, word answer and word matching.

    Diagnostic Assessment Examples for Psychology

    The professor may conduct a survey in order to evaluate assumptions students currently hold about concepts like the nature of the mind versus human behavior.

    In psychology or sociology courses dealing with controversial or sensitive topics, instructors may conduct student surveys to allow learners to pose questions or potentially controversial viewpoints anonymously, allowing for more open classroom discussions and more thorough understandings of preconceived notions students might hold. 

    Examples of Diagnostic Assessment in Psychology Tools

    Socrative is a quiz and assessment website that lets instructors design interactive quizzes particularly suitable for complex topics in psychology, like bio-psychology, criminological psychology, statistics and research methods.

    D2L lets instructors create several types of diagnostic assessments for psychology, including quizzes, surveys and self-assessments.

    Diagnostic Assessment Examples for Creative and Fine Arts

    Instructors can also use pre-assessment and self-assessment tests to help better direct their effort to inspire their students to engage with class material by seeing what students already comprehend about the complexities of the creative process. They can also collect initial portfolios to judge fine-arts students’ artistic abilities while simultaneously conveying the course objectives.

    Examples of Diagnostic Assessment in Education in Creative Arts and Fine Arts Tools

    Besides allowing professors to create customized short-form quizzes, Canvas Quizzes also contains a special “Assignments” feature that lets students upload a file for assessment. This can include a piece of creative written, illustrated or even audio/visual material. That flexibility of media allows professors to examine a broader range of skills and competencies than can be assessed through simple question and answer assessments alone.

    Diagnostic Assessment Examples for STEM courses

    More than other subjects, math can create a particularly large amount of anxiety in students who struggle with the subject, yet it can be significantly more difficult for instructors to target math interventions for students. If math anxiety and issues with math aren’t properly identified and targeted soon enough, however, they could easily escalate into much more deeply-rooted learning problems even more challenging for students to overcome.

    Diagnostic assessments help professors gauge students’ current level of competency in complex problem-solving in a number of prerequisite areas before beginning to teach them concepts intended to build upon that knowledge. This may include basic algebraic manipulations, cell cycles, solving equations and chemical equations. By implementing data-driven approaches, professors can specifically examine how students think about math and what strategies and skills they bring with them to approach a math problem.

    An effective diagnostic assessment for math typically examines only one skill or set of skills at a time. That way, professors can more easily identify areas and concepts where students may be in need of further review.

    Tools for Diagnostic Assessments in STEM courses

    Top Hat offers a suite of secure tests and exam features that allow instructors to create diagnostic assessments for both in-person and online learning settings with equal ease and efficiency. Whether remote and proctored diagnostic assessment for math or on-premise and open book diagnostic assessment for math, Top Hat’s secure tests and exams feature lets you choose from 14 different question types or access the Top Hat Catalog and select from a variety of pre-made test banks for mathematics diagnostic assessment.

    For online testing, you can verify identities and devices, monitor activity and receive reports flagging irregular behavior. You can create, deploy and receive exams all in one place and have the exams auto-graded. Helping make mathematics diagnostic assessment easier, you can also customize question types and assignment settings and you can let students upload mathematics diagnostic assessment projects as PDF files, spreadsheets and slide presentations.

    Key Examples of Diagnostic Assessments

    Unit Pretests

    Unit pretests are a type of diagnostic evaluation tool that does not involve students receiving any grades. Instead, unit pretests are a diagnostic test in education example of how to determine a student’s awareness of a certain unit or module of learning within a larger course before proceeding to learn it. This type of diagnostic test in education example may include multiple-choice or fill-in-the-blank questions, as opposed to those of a more open-ended nature. For best use of these examples of a diagnostic test in education, unit pretests are most effective when concentrated on the core skill or concept for students to understand rather than the finer minutiae of the subject matter.

    Exit Tickets

    Exit tickets are a straightforward example of how to most effectively gauge student understanding after teaching a lesson, when you’re looking to see how effectively your students have met the objectives for that lesson or unit.

    Instructors ask students a simple question relating to a key concept taught in the lesson they’ve just concluded. Students jot down their answers on a “ticket” they deliver to the instructor upon their “exit” from the classroom. This allows instructors to adapt and adjust their curriculum for the following lesson or semester to align actual exit ticket results more closely with desired outcomes.

    Conclusion

    Diagnostic assessment examples like these provide instructors insights that help them to better create curricula customized to their students’ current level of knowledge, skills, strengths and weaknesses and, thereby, to better aid their students in achieving the objectives of the course. Likewise, professors can apply examples of diagnostic assessment in education like these after teaching a lesson or course in order to determine how well the objectives for that lesson or course were met and, based on that information, better strategize and adapt the curriculum for the next lesson or course.

    As these diagnostic assessment examples show, diagnostic evaluations are generally informal and simple to use. They typically require no high-level training to create and don’t require following any standardized protocol. Instructors can alter or more finely tune their assessment methods any time they wish. Instructors can share what they discover through the various types of diagnostic assessments they use with their peers quickly and easily. These examples of diagnostic tests in education and others like them work for any discipline and, most importantly, once applied with the right tools and technologies, diagnostic assessments in education show fast and efficient results.

    Tagged as:

    ,

    Source link