Taccle AI

Improving skills and competences of VET teachers and trainers in the age of Artificial Intelligence

Assessment

News 

Considerations for Curriculum and Assessment design

Graham Attwell 34 Views 0 Comments , , , , , , ,

Drawing on the findings of the GENIAL project, which focused on how generative AI tools are used in practice by students in real time, Dorottya Sallai, Jon Cardoso-Silva and Marcos Barreto analyse how students use these tools differently across qualitative and quantitative subjects and offer recommendations for how educators can integrate these findings into their teaching and assessment plans in a blog article, To improve their courses, Educators should respond to how students actually use AI, on the London School of Economics website.

They say their study suggests that students rely more on AI tools when they are struggling with the quantity of the reading materials or the complexity of the course content and tend to lean less on it when the pace of delivery and the content is more accessible to them. As one of the management students expressed it: “This week’s content was pretty straightforward, and I haven’t found myself using AI”

Based on these preliminary findings, they set out some practical policy recommendations for university educators in relation to course and assessment design. Given the high likelihood that students may use AI as shortcuts, they say educators must find strategies to ensure that students use these tools to enhance their learning rather than bypassing it.

They go on to propose practical Curriculum Design considerations and considerations for Assessment Design.

Curriculum Design Considerations
- Assume Student Use of GenAI: Plan with the expectation that students will use GenAI tools.
- Integrate Non-Marked Activities: Include activities that are not graded but provide feedback on AI use.
- Ensure Full Engagement: Prevent GenAI from diminishing students' engagement with the curriculum. Prepare students to progress beyond AI-generated solutions.
- Teach Critical Analysis: Emphasize the importance of finding primary sources and critically evaluating AI outputs.
- Avoid Underspecified Assignments: Do not attempt to outsmart AI by underspecifying tasks, as future models may overcome these tactics.
- Coding Course Guidance: Instruct students on problem identification and correction in coding. Highlight alternative solutions and teach high-level engineering concepts by analyzing and improving AI outputs.

Assessment Design Considerations
- Process Mapping: Visualize the learning journey with milestones and check-in points to evaluate students’ progress.
- Separate Learning from Final Product: Design continuous assessments throughout the term or incorporate documentation of the development process in end-of-course evaluations.
- Measure Individual Learning: Use in-class quizzes at various stages of the term to gauge and support individual student progress. Use these assessments to benchmark final grades against the students' learning journeys

News 

Homework Apocolypse?

Graham Attwell 41 Views 0 Comments , , , , , , ,
Catherine Breslin & Tania Duarte / Better Images of AI / AI silicon clouds collage / CC-BY 4.0

November marks two years since the release of Open AI's GPT large language model chatbot. Since then AI, or more specifically Generative AI has dominated the discourse over the future of education. And of course it has spawned hundreds of project resulting in an increasing torrent of research results. Yet on one critical issue - does the use of AI improve learning - there appears little consensus. This is probably because we have no good ways of measuring learning. Instead we use performance in tests and exams as a proxy for learning. And its probably true to say that the debates over AI are turning the heat on the use of such a proxy, just as it is on the essay as the dominant form of assessment in schools and universities.

Last week in his newsletter, One Useful thing, Ethan Mollick talked about the use of AI, cheating and learning in an article entitled 'What comes after the Homework Apocalypse'. It is probably fair to say Ethan is a big fan of AI in education.

To be clear, AI is not the root cause of cheating. Cheating happens because schoolwork is hard and high stakes. And schoolwork is hard and high stakes because learning is not always fun and forms of extrinsic motivation, like grades, are often required to get people to learn. People are exquisitely good at figuring out ways to avoid things they don’t like to do, and, as a major new analysis shows, most people don’t like mental effort. So, they delegate some of that effort to the AI. In general, I am in favor of delegating tasks to AI (the subject of my new class on MasterClass), but education is different - the effort is the point.

He postulated that fall in grades achieved by students in the USA between 2008 and 2017 had been caused by the increasing use of the Internet for homework. Students were simply copying homework answers. And in an experiment in ma high school in Turkey with students using GPT4 grades for homework went up but final exam grades fell. But giving students GPT with a basic tutor prompt for ChatGPT, instead of having them use ChatGPT on their own, boosted homework scores without lowering final exam grades. 

Ethan says this shows "we need to center teachers in the process of using AI, rather than just leaving AI to students (or to those who dream of replacing teachers entirely). We know that almost three-quarters of teachers are already using AI for work, but we have just started to learn the most effective ways for teachers to use AI."

He remains convinced to the value of Generative AI in education. The question now, he says "is not whether AI will change education, but how we will shape that change to create a more effective, equitable, and engaging learning environment for all."

News 

AI: What do teachers want?

Graham Attwell 57 Views 0 Comments , , , , , , , ,
Yutong Liu & Kingston School of Art / Better Images of AI / Talking to AI / CC-BY 4.0

A quick post in follow up to my article yesterday on the proposals by the UK Department for Education to commission tech companies to develop an AI app for teachers to save them time. The Algorithm - a newsletter from MIT Technology Review picked up on this today, saying "this year, more and more educational technology companies are pitching schools on a different use of AI. Rather than scrambling to tamp down the use of it in the classroom, these companies are coaching teachers how to use AI tools to cut down on time they spend on tasks like grading, providing feedback to students, or planning lessons. They’re positioning AI as a teacher’s ultimate time saver."

The article goes on to ask how willing teachers are to turn over some of their responsibilities to an AI model? The answer, they say, really depends on the task, according to Leon Furze, an educator and PhD candidate at Deakin University who studies the impact of generative AI on writing instruction and education.

“We know from plenty of research that teacher workload actually comes from data collection and analysis, reporting, and communications,” he says. “Those are all areas where AI can help.”

Then there are a host of not-so-menial tasks that teachers are more skeptical AI can excel at. They often come down to two core teaching responsibilities: lesson planning and grading. A host of companies offer large language models that they say can generate lesson plans that conform to different curriculum standards. Some teachers, including in some California districts, have also used AI models to grade and provide feedback for essays. For these applications of AI, Furze says, many of the teachers he works with are less confident in its reliability. 

Companies promising time savings for planning and grading “is a huge red flag, because those are core parts of the profession,” he says. “Lesson planning is—or should be—thoughtful, creative, even fun.” Automated feedback for creative skills like writing is controversial too. “Students want feedback from humans, and assessment is a way for teachers to get to know students. Some feedback can be automated, but not all.” 

News 

LLMs are a cultural technology

Graham Attwell 60 Views 0 Comments , , , , , ,
Yutong Liu & Kingston School of Art / Better Images of AI / Exploring AI / CC-BY 4.0

John Naughton writing in the Guardian says:

Assessment in humanities in time of LLMs requires, "if not a change of heart, two changes of mindset.

The first is an acceptance that LLMs – as the distinguished Berkeley psychologist Alison Gopnik puts it – are “cultural technologies”, like writing, print, libraries and internet search. In other words, they are tools for human augmentation, not replacement.

Second, and more importantly perhaps, is a need to reinforce in students’ minds the importance of writing as a process."

News 

GenAI and Assessment

Graham Attwell 100 Views , , , , , , ,

As a recent publication from the Universitat Oberta de Catalunya points out, Artificial Intelligence remains an opportunity (or an excuse) to transform assessment, curriculum, teaching, personalization and teaching competencies. This is especially so in relation to assessment with widespread concern in the academic world about the near impossibility of detecting whether or not a student has used generative AI in an assignment.

The Universitat Oberta de Catalunya article explores the potential of continuous assessment aimed at self-regulation of learning. It suggests changing the assessment approach, moving from criteria focused on the assessment of the result to criteria focused on the process of development of the activity by the students.

    Furthermore it advocates designing continuous assessment activities as part of the same learning sequence, with relationships of dependency and complementarity, instead of discrete tests and focusing the activities on the development of competencies and the assessment of progress and reflection on the learning process of each student.

    Leon Furze is a prolific contributor to LinkedIn and describes his work as "Guiding educators through the practical and ethical implications of GenAI. Consultant & Author | PhD Candidate."

    Witting from the perspective of education in Australia he says:

    When it comes to GenAI, much of the conversation in education has been focused on academic achievement, perceived threats to academic integrity, and the risk that this technology poses to written assessments. I think that vocational education actually offers some fantastic alternative forms of assessment which are less vulnerable to generative artificial intelligence. If you’re not familiar with vocational education, assessments are often incredibly rigorous, sometimes to the point where the paperwork on evaluation and assessment is significantly longer than the assessment itself.

    Vocational training, by nature, is practical and geared around skills which are needed for the particular job role or discipline being studied. Mainstream education, by contrast, is focused predominately on subjects and content.

    Furze provides examples of different types of assessment in vocational educati9n and training:

    • Observation checklists
    • Role plays
    • Scenarios
    • Workplace activities
    • Reports from employers

    He has prublished a free 60 page ebook - Rethinking Assessment for GenAI which he says covers everything from ways to update assessments, to the reasons I advise against AI detection tools.