Comparative Evaluation of ChatGPT and Students in the Outcomes of Online Learning Courses Related to Security

Artificial Intelligence has increasingly affected the educational scenario, and its careful assessment is more and more essential to avoid misuses and not fair exploitation on the part of the students. This paper provides a preliminary comparison between the performance of a very well-known Large Language Model, namely ChatGPT 3.5, and those of a cohort of students attending online courses about network and computer security in an online university. The analysis has been performed by considering both multiple-choice as well as open-ended questions and did not consider any form of fine-tuning nor contextualization for the training of the considered Large Language Model. The obtained results demonstrate a very good performance of ChatGPT 3.5 in answering the prompted questions on both courses and on both types of questions. Moreover, we have also analyzed some of the few hallucinations that took place whenever the overall context of the questions was very similar or even the same.