AI Procurement: key questions

Alexa Steinbrück / Better Images of AI / Explainable AI / CC-BY 4.0

In the AI pioneers project we are frequently asked by teachers. and. trainers in Vocational Education and Training and Adult Education what they should be looking for if they intend licensing or buying AI based applications. The UK Jisc has developed and published an AI Maturity model. "As institutions move to the ’embedded’ stage," they say "we expect appropriate processes to be in place for the entire lifecycle of AI products, including procurement."

They continue to explain that: "This detailed scrutiny aims to facilitate a better understanding and mitigation of potential risks associated with AI deployment. Additionally, it is crucial to ensure that the use of AI in educational and research settings does not infringe on IP rights and that the data used in AI models is appropriately managed to maintain research integrity and protect proprietary information."

The model includes comprehensive due diligence processes for areas such as supplier information, financial stability, insurance coverage, modern slavery compliance, information security, and data protection. By thoroughly vetting these aspects, JISC says, we aim to ensure that any solutions are not only innovative and effective but also ethical and compliant with all relevant regulations and standards. The questions are intended to be dynamic and will be reviewed to reflect advances in technology or legislation.

1Outline which AI features of your system use third party AI models, and which use your own proprietary or in-house AI models.  Please provide details of any third-party technologies used, including the name of provider and an outline of the features used.  Note that for major suppliers in the LLM supply chain, such as OpenAI, Google DeepMind, Anthropic, etc., due diligence should be conducted separately. There’s no need to request information about them from all third-party providers built on these large language models.
2Where you are either creating your own model or fine tuning a third-party model, how is performance defined and measured? Include details of initial training and monitoring over time.  (UK AI Principle: Safety, security and robustness)
3What data do your AI models require for initial training or fine tuning? If you are using third party models, you should only describe data that is unique to your application.  (UK AI Principle: Safety, security and robustness)
4a/4bIs data from user interactions with the system utilized to enhance model performance, and if so, please elaborate on the mechanisms involved? Furthermore, could you provide clarification on whether institutional data is integrated into external models?  (UK AI Principle: Safety, security and robustness)
5What features does your solution have to make it clear when the user is interacting with an AI tool or AI features?  (UK Principle: Safety, security and robustness)
6Could you please provide comprehensive information about the safety features and protections integrated into your solution to ensure safe and accessible use by all users, including those with accessibility needs and special education requirements?(UK Principle: Safety, security and robustness) 
7Can you specify any special considerations or features tailored for users under the legal majority age?UK Principle: Safety, security and robustness) 
8What explainability features does your AI system provide for in its decisions or recommendations?(UK Principle: Safety, security and robustness) 
9What steps are taken to minimize bias within models your either create or fine tune?(UK Principle: Fairness robustness) 
10Does your company have a public statement on Trustworthy AI or Responsible AI? Please link to it here.(UK Principle: Accountability and governance) 
11/ 11a/ 11b/ 11c Does your solution promote research, organizational or educational use by: A)    Not restricting the use of parts of your solution within AI tools and services B)    Not preventing institutions from making licensed solutions fully accessible to all authorized users in any legal manner; C)    Not introducing new liability on institutions, or require an institution to indemnify you especially in relation to the actions of authorized users (Gartner, Inc, ICOLC statement and legal advice obtained by Jisc)
12Does your solution adequately protect against institutional intellectual property (IP) infringement including scenarios where third parties are given access to and may harvest institutional IP?(Gartner, Inc and ICOLC statement)

A Compassionate Approach to AI in Education

Alina Constantin / Better Images of AI / Handmade A.I / CC-BY 4.0

I very much like this blog post, A Compassionate Approach to AI in Education, by Maha Bali from the American University in Cairo. Maha explains where she is coming from. And she addresses ethics, not from the point of an abstract ethical framework, of which we have many at the moment, but from the point of ethical practice. What follows is a summary but please read the whole blog

The article discusses the challenges and opportunities that generative artificial intelligence (AI) presents in education, from the viewpoint of a teacher and researcher deeply involved with educators worldwide through these changes. She emphasises a feminist approach to education, centered on socially just care and compassionate learning design, which critically examines the inequalities and biases exacerbated by AI technologies. The article is structured around four key strategies for educators and learners to adapt and respond to AI's impact:

  1. Critical AI Literacy: Developing an understanding of how AI operates, especially machine learning, is fundamental. Educators and students must grasp how AI outputs are generated, how to judge their quality, and where biases might be embedded. Training data for AI, often dominated by Western, white, and male perspectives, can reinforce existing inequalities, particularly affecting underrepresented groups. The author provides an example where an AI tool incorrectly associated an Egyptian leader with an unrelated American figure, highlighting the importance of recognising biases and inaccuracies. The global South is often underrepresented in training data, and the AI workforce is predominantly male, which can discourage women from pursuing technical skills.
  2. Appropriate AI Usage: While some AI uses have proven beneficial, such as medical diagnostics and accessibility features for visually impaired people, educators must distinguish when its application could be harmful or unethical. AI's biases and limitations mean it should not be relied upon for personalised learning or critical assessments. The EU has identified high-risk AI applications that require careful regulation, including facial recognition and recruitment systems. In educational settings, AI should not replace human judgment in crucial evaluations, and the emotional aspects of learning should not be overlooked.
  3. Inclusive Policy Development: Students should be actively involved in shaping AI policies and guidelines within classrooms and institutions. The author suggests using metaphors to help learners understand when AI is appropriate, comparing it to baking a cake. For instance, sometimes students need to bake a cake from scratch (doing all work without AI), while other times, they can use pre-made mixes (using AI as a starting point) or purchase a cake (fully using AI). By having these discussions, students understand the purpose of assignments and when AI can enhance or detract from learning outcomes.
  4. Preventing Unauthorized AI Use: Understanding why students might be tempted to use AI unethically is critical. Students often misuse AI due to tight deadlines, lack of interest or understanding in assignments, lack of confidence in their abilities, and competitive educational environments. The author advocates for empathetic listening, flexible deadlines, and creative assignments that encourage genuine engagement. Moreover, fostering a supportive classroom community can reduce competitiveness and emphasise collaborative learning over competition.

The article encourages a compassionate, critical approach to AI in education. By understanding the biases embedded in AI, developing critical AI literacy, and involving students in policy-making, educators can ensure that students ethically and effectively use AI tools. This approach aims to empower learners to shape future AI platforms and educational systems that are socially just and inclusive.