Dalarna University's logo and link to the university's website

Teachers in higher education often face issue with enormous emails from student regarding academic queries. However, many of these questions are repetitive and can be found in course handbook which is already uploaded on the website. This redundant communication load not only consumes valuable time of teachers also sometimes student may not get the responses on time. This thesis explores the use of Artificial Intelligence (AI) agents particularly Large Language Models (LLMs) as a mediator between student-teacher communication by answering to the student questions by solely based on course manual.

This study assesses two modern LLMs, OpenAI’s GPT-4o-mini (OpenAI, 2024) and Meta’s LLAMA 3.2 (Di Palma et al., 2024). A balanced selection of both answerable and unanswerable questions was included in each of the five course handbooks from a recognized academic field. To highlight different models’ behaviours, final accuracy comparisons concentrated on Prompt 1(loose) and Prompt 3(strict), although a three-stage prompt design (White et al., 2023) was evaluated. A Likert-scale evaluation (1-3) was used to validate the model’s outputs, which were then categorized into (TP, TN, FN, FP) and confirmed by a combination of manual and LLM assisted review.

The results revealed that LLaMA 3.2 achieved highest accuracy with strict prompt, with the lowest average errors rates (FN and FP) and the highest average correct responses (TP+TN) while GPT-4o-mini remained prone errors. This emphasise how important prompt clarity and instruction tuning (Raina, Liusie, & Gales, 2024) are. Although, encouraging the study’s scope is constrained by its manual review and handbook structure, indicating the necessity of more extensive validation and automation in future work.