In last weeks edition of her newsletter, Philippa Hardman reported on an interesting research project she has undertaken to explore the effectiveness of Large Language Models (LLMs) like ChatGPT, Claude, and Gemini in instructional design. It seems instructional designers are increasingly using LLMs to complete learning design tasks like writing objectives, selecting instructional strategies and creating lesson plans.
The question Hardman set out to explore was: “how well do these generic, all-purpose LLMs handle the nuanced and complex tasks of instructional design? They may be fast, but are AI tools like Claude, ChatGPT, and Gemini actually any good at learning design?” To find this out she set two research question. The first was sound the Theoretical Knowledge of Instructional Design by LLMs and the second to assess their practical application.She then analysed each model’s responses to assess theoretical accuracy, practical feasibility, and alignment between theory and practice.
In her newsletter Hardman gives a detailed account of the outcomes of testing the different models from each of the three LLM providers, But the The headline is that across all generic LLMs, AI is limited in both its theoretical understanding and its practical application of instructional design. The reasons she says is that they lack industry specific knowledge and nuance, they uncritically use outdated concepts and they display a superficial application of theory.
Hardman concludes that “While general-purpose AI models like Claude, ChatGPT, and Gemini offer a degree of assistance for instructional design, their limitations underscore the risks of relying on generic tools in a specialised field like instructional design.”
She goes on to point out that in industries like coding and medicine, similar risks have led to the emergence of fine-tuned AI copilots, such Cursor for coders and Hippocratic AI for medics and sees a need for “similar specialised AI tools tailored to the nuances of instructional design principles, practices and processes.”
In last weeks edition of her newsletter, Philippa Hardman reported on an interesting research project she has undertaken to explore the effectiveness of Large Language Models (LLMs) like ChatGPT, Claude, and Gemini in instructional design. It seems instructional designers are increasingly using LLMs to complete learning design tasks like writing objectives, selecting instructional strategies and creating lesson plans.
The question Hardman set out to explore was: “how well do these generic, all-purpose LLMs handle the nuanced and complex tasks of instructional design? They may be fast, but are AI tools like Claude, ChatGPT, and Gemini actually any good at learning design?” To find this out she set two research question. The first was sound the Theoretical Knowledge of Instructional Design by LLMs and the second to assess their practical application.She then analysed each model’s responses to assess theoretical accuracy, practical feasibility, and alignment between theory and practice.
In her newsletter Hardman gives a detailed account of the outcomes of testing the different models from each of the three LLM providers, But the The headline is that across all generic LLMs, AI is limited in both its theoretical understanding and its practical application of instructional design. The reasons she says is that they lack industry specific knowledge and nuance, they uncritically use outdated concepts and they display a superficial application of theory.
Hardman concludes that “While general-purpose AI models like Claude, ChatGPT, and Gemini offer a degree of assistance for instructional design, their limitations underscore the risks of relying on generic tools in a specialised field like instructional design.”
She goes on to point out that in industries like coding and medicine, similar risks have led to the emergence of fine-tuned AI copilots, such Cursor for coders and Hippocratic AI for medics and sees a need for “similar specialised AI tools tailored to the nuances of instructional design principles, practices and processes.”
George Bekiaridis and Graham Attwell have made a keynote presentation to the Second Conference on the Reference Framework of Competences for Democratic Culture and Vocational Education and Training to be held on 24 and 25 October 2024 at the Council of Europe in Strasbourg, France. The event was be dedicated to discussing the chapters of the new publication on the Council of Europe’s Reference Framework of Competences for Democratic Culture (RFCDC) and VET.
In the presentation, Transforming Vocational Training - AI in theory and practice, they introduced ongoing research on using Activity Theory to analyse the impact of AI learning as a result of tool-mediated interactions, showcasing how conceptual frameworks, technologies, practical actions, individuals, and social institutions mutually shape each other in the learning process. They drew attention to the UNESCO Framework for competences in AI for students. which emphasises the importance of competences for citizenship, similar to the work of the Council of Europe's work on Democratic Culture. You can download a copy of the presentation here. It is licensed under a Creative Commons Creative Commons Attribution-NonCommercial 4.0 International License.
Yutong Liu & Kingston School of Art / Better Images of AI / Talking to AI 2.0 / CC-BY 4.0
There's an interesting post from Philippa Hardman in her newsletter today. Entitled Are ChatGPT, Claude & NotebookLM *Really* Disrupting Education? her research asks how much and how well do popular AI tools really support human learning and, in the process, disrupt education? She created a simple evaluation rubric to explore five key research questions:
1. Inclusion of Information
2. Exclusion of Information
3. [De]Emphasis of Information
4. Structure & Flow
5. Tone & Style
Philippa Hardman used her own research articles as the input material, which she fed into what she says are considered to be the three big AI tools for learning:
She prompted each tool in turn to read the article carefully and summarise it, ensuring that it covered all key concepts, ideas etc ensuring that I get a thorough understanding of the article and research.
She provides a detailed table of the results of each of the three applications, and additionally of the NotebookLM podcast application, assessing the strengths and weaknesses of each. she says that "while generative AI tools undoubtedly enhance access to information, they also actively “intervene” in the information-sharing process, actively shaping the type and depth of information that we receive, as well as (thanks to changed in format and tone) its meaning. "
She goes on to say:
While popular AI tools are helpful for summarising and simplifying information, when we start to dig into the detail of AI’s outputs we’re reminded that these tools are not objective; they actively “intervene” and shape the information that we consume in ways which could be argued to have a problematic impact on “learning”.
Another thing is also clear: tools like ChatGPT4o, Claude & Notebook are not yet comprehensive “learning tools” or “education apps”. To truly support human learning and deliver effective education, AI tools need to do more than provide access to information—they need to support learners intentionally through carefully selected and sequenced pedagogical stages.
Her closing thoughts are about Redefining the “Learning” Process . She says:
It’s clear that AI tools like ChatGPT, Claude, and NotebookLM are incredibly valuable for making complex ideas more accessible; they excel in summarisation and simplification, which opens up access to knowledge and helps learners take the first step in their learning journey. However, these tools are not learning tools in the full sense of the term—at least not yet.
By labelling tools like ChatGPT 4o, Claude 3.5 & NotebookLM as “learning tools” we perpetuate the common misconception that “learning” is a process of disseminating and absorbing information. In reality, the process of learning is a deeply complex cognitive, social, emotional and psychological one, which exists over time and space and which must be designed and delivered with intention.
Here's the follow up I promised in my last post about earners' and teachers' Agency and Gen AI.
Motivation plays a crucial role in the learning process. As opposed to behaviorist theories of learning, learners are increasingly seen as active participants in learning leading to a focus on how learners make sense of and choose to engage with their learning environments (National Academies of Sciences, Engineering, and Medicine. 2018). Cognitive theories, for example, have focused on how learners set goals for learning and achievement and how they maintain and monitor their progress toward those goals. While earlier research focused largely on the classroom environment, newer research, especially following the emergency online learning turn during the Covid19 emergency, has looked at the online learning environment mediated by various forms of technology (Thomas K. F. Chiu, Tzung-Jin Lin, and Kirsti Lonka. 2021) Social interactions mediated by technology affect learning through their impacts on students’ goals, beliefs, affect, and actions (Social interactions mediated by technology affect learning through their impacts on students’ goals, beliefs, affect, and actions (Manjur Kolhar, Raisa Nazir Ahmed Kazi, Abdalla Alameen, 2021).
“Motivation is also increasingly viewed as an emergent phenomenon, meaning it can develop over time and change as a result of one’s experiences with learning and other circumstances” (Thomas K. F. Chiu, Tzung-Jin Lin, and Kirsti Lonka, 2021) . Research suggests, for example, that aspects of the learning environment can both trigger and sustain a student’s curiosity and interest in ways that support motivation and learning (Hidi and Renninger, 2006). Of course the converse can apply with learning environments reducing motivation.
There are an increasing number of studies looking at how Generative AI impacts on learning and motivation. Yet many of these are attempting to measure the effectiveness of learning and rely on achievement in assessment as a proxy for learning. Excepting learning to program, there is limited evidence from Vocational Education and Training, despite VET being largely learning outcomes based. However, measuring effectiveness and motivation in VET is made more complicated by the many different models of VET provision.
Neither is there any consensus about the efficacy of AI for learning. In his OL Daily Newsletter, Stephen Downes (2024) discusses a LinkedIn post from Ethan Mollick stating "AI can help learning... when it isn't a crutch." Mollick cites three papers: firstly AI Meets the Classroom: When Does ChatGPT Harm Learning? which states "Using LLMs as personal tutors by asking them for explanations improves learning outcomes whereas excessively asking LLMs to generate solutions impairs learning." Second, Generative AI Can Harm Learning says "students attempt to use GPT-4 as a 'crutch' during practice problem sessions, and when successful, perform worse on their own" though "These negative learning effects are largely mitigated by the safeguards included in GPT Tutor." Third, Effective and Scalable Math Support says "chat-based tutoring solutions leveraging AI could offer a cost-effective and operationally efficient approach to enhancing learning outcomes for millions of students globally." Downes concludes “All these results are, at worst, mixed, and at best, show genuine promise in AI for improving learning.”
Of course, motivation is only one factor in improving learning. Motivation is generally divided between Intrinsic and extrinsic motivation. Generative AI can potentially enhance intrinsic motivation through immediate feedback and adaptive challenges and enabling more creative and open-ended projects that align with students' interests It can also offer novel and engaging ways to interact with learning materials. But Artemova (2024) says “it has been demonstrated that students are primarily involved with learning activities for reasons other than epistemological curiosity or a desire to learn. Instead, they are motivated by the desire to interact with technology or to meet the expectations set by educational software.”
And while extrinsic motivation can be effective, over-reliance on AI-powered reward systems or gamification elements may lead to a focus on external rewards rather than the inherent value of learning. There is also a danger that students might become overly dependent on AI assistance, potentially undermining their confidence in their own abilities and the ease of generating content with AI might lead to questions about the authenticity of student work, potentially impacting intrinsic motivation. A further concern is that the availability of instant AI-generated answers might reduce students' motivation to engage in effortful cognitive processes.
Inna Artemova (2024) who has undertaken an analysis of 69 articles for her paper Digital Education Review ‘Bridging Motivation and AI in Education: An Activity Theory Perspective’ concludes “that in 56 research papers motivation is seen as extrinsic, which implies a greater involvement of students in the learning process due to increased interactivity and adaptability of the content (Yang et al., 2020). Through text analysis, it is clear that this type of motivation is driven by motives-stimuli, such as personalised learning environments (Bulathwela et al., 2024), which in fact means that motivation in this case is secondary to the AI implementation and is guided by the AI.”
If I may add a personal viewpoint drawn from my own learning of Spanish using the popular DuoLingo online learning environment, which is heavily gamified and provides personalised learning content, it develops both intrinsic and extrinsic motivation. Particularly effective is the exhortation to practise regularly using the idea of a ‘streak’ based on how many continuous days you have accessed the application (although it also allows a limited streak freeze. I have now been learning on DuoLingo for a three year streak. How effective my learning has proved to be is another question.
Clearly, as with so much on AI and education, this is an emergent area of research with contested viewpoints. But we would tentatively conclude that while Generative AI offers many opportunities to enhance motivation, it may also present challenges that need to be addressed. Educators must be aware of these potential pitfalls and develop strategies to maintain healthy motivational patterns in AI-enhanced learning environments.
Bulathwela, S., Pérez-Ortiz, M., Holloway, C., Cukurova, M., & Shawe-Taylor, J. (2024). Artificial Intelligence Alone Will Not Democratise Education: On Educational Inequality, Techno-Solutionism and Inclusive Tools. Sustainability, 16(2), 781. https://doi.org/10.3390/su16020781
Downes, S. (2024) Student use of LLMs can inhibit learning, https://www.downes.ca/post/77127
Henkel, H., 1, Horne-Robinson, H., Kozhakhmetova, N., Lee, A. Effective and Scalable Math Support: Experimental Evidence on the Impact of an AI- Math Tutor in Ghana . https://arxiv.org/ftp/arxiv/papers/2402/2402.09809.pdf
Kolhar, M., Nazir R., Kazi, A., Alameen, A. (2021) Effect of social media use on learning, social interactions, and sleep duration among university students, Saudi Journal of Biological Sciences, Volume 28, Issue 4,
Matthias Lehmann,M., Cornelius, P., Sting F. (2024) AI Meets the Classroom:
Mollick, E. (2024) https://www.linkedin.com/posts/emollick_ai-can-help-learning-when-it-isnt-a-crutch-activity-7250556786640924672-Enhg/
National Academies of Sciences, Engineering, and Medicine. 2018. How People Learn II: Learners, Contexts, and Cultures. Washington, DC: The National Academies Press. https://doi.org/10.17226/24783.
Thomas K. F. Chiu, Tzung-Jin Lin, and Kirsti Lonka. (2021) Motivating Online Learning: The Challenges of COVID-19 and Beyond, https://link.springer.com/content/pdf/10.1007/s40299-021-00566-w.pdf
Yang, D., Oh, E.-S., Wang, Y. (2020). Hybrid Physical Education Teaching and Curriculum Design Based on a Voice Interactive Artificial Intelligence Educational Robot. Sustainability,12(19), 8000. https://doi.org/10.3390/su12198000
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPT
Privacy & Cookies Policy
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
