SokratesT

Under certain conditions, AI chatbots can enhance personalised learning, improve learning outcomes and learner motivation, and boost learners’ self-efficacy and sense of competence.

Despite this potential, however, unregulated use can have potentially serious implications for the effectiveness and legality of teaching and assessment practices. For example, if students upload documents or textbooks to ChatGPT, this may give rise to copyright issues. Uninformed use of AI can even impair performance, without the tools necessarily having to ‘hallucinate’. Thus, even factually correct AI content can diminish learning success if it emphasises irrelevant points or incorrectly links aspects. Furthermore, AI tempts students to have answers generated for them without recognising connections, being able to apply what they have learnt to other problems, or synthesising knowledge – all of which are essential steps in the acquisition of competences according to Bloom/Krathwohl.

The question is how the creative potential of generative AI can be reconciled with data protection, teaching methodology and examination regulations, whilst simultaneously ensuring the quality of course-relevant content. The aim is to create a bot that helps students explore content at their own pace, based on their prior knowledge and interests, within the framework of a specific course with defined learning objectives and examination requirements. This requires a clearly defined, validated corpus of information. To this end, we employ a method that has been tried and tested for years: Retrieval Augmented Generation (RAG). In this process, the language model’s responses are largely based on a database fed by previously validated knowledge graphs, documents and media.

The challenge is not merely technical: a factually correct answer is only valuable from a didactic perspective if it is integrated into an analytical thought process that goes beyond mere reproduction. It would be more effective if the AI, through a Socratic dialogue involving targeted questions, encouraged learners to question gaps in their knowledge, rather than simply spitting out the supposedly correct answer. Socratic dialogue has proven its worth in higher education for sharpening critical thinking and lends itself well as a fundamental pedagogical principle for an AI tutor.

The Sokratest Methodology
is being developed collaboratively with students and lecturers. To this end, we are adopting an agile approach, iterating through four development phases based on feedback. Feedback from stakeholders during these phases allows for flexible development without losing sight of the project’s objective. 

To begin with, the technical framework for the RAG bot and the design of the process model will be developed. In parallel, a complementary psychological study will be developed to investigate expectations and experiences. 

We are testing how the data corpus and language model can best be combined and analysing aspects such as performance, prevention of manipulation, didactics and data protection. The code will be made available to interested users via GitHub. In addition, we are developing a process model and survey tool as an Open Educational Resource (OER), so that potential users can automatically generate a bot tailored to their needs.

The data corpus and research questions are defined within the module, after which the real-world laboratory begins. We expect to see an increase in the participants’ intrinsic motivation and AI literacy, and will analyse the pseudonymised queries and dialogues accordingly. Once we have a sufficient data set, we will jointly consider certain adjustments, such as to the response style. 

Usage data will be systematically evaluated and the development phases documented. A process model will be developed, enabling internal and external transfer to begin (see 1.4). Phase 4 concludes with the analysis and preparation of the results for discussion within the university, which in turn provides the basis for scaling up the project. 

For copyright reasons, it is assumed that no large-scale language models subject to licensing fees (e.g. LLMs such as ChatGPT) will be used, but rather local open-source solutions for smaller language models. Particularly when applying RAG to protected content, open-source models (e.g. Mistral’s ‘Mixtral’) offer greater flexibility and can be run more securely. As the anticipated knowledge corpus will include not only texts but also audiobooks, videos and other media formats, multimodal aspects will be identified and utilised in data analysis and, ideally, also in the presentation of results and interaction with the bot. 

Scalability, Utilisation and Transfer
To promote scaling at an early stage, internal and external information sessions are already planned. The student body was closely involved in the design of a comprehensive survey on AI usage. Building on this positive experience, we are also planning a systematic exchange with student representatives here, with the aim of making students a driving force behind scaling up the initiative. The starting point is the pilot implementation in two English-language course modules: Business Administration (Bachelor’s) and Business Psychology (Master’s).