Evaluating the Success of AI Tools in Supporting Student Performance in Mathematical Kangaroo Competition

Abstract

This study explores the potential of generative artificial intelligence (AI) tools in supporting students preparing for mathematical competitions, focusing on the Mathematical Kangaroo competition in the context of the Serbian-speaking region. The research analyzed tools such as ChatGPT-free, ChatGPT-paid, AI Math Solver, Math Mentor, and o1-preview, assessing their accuracy and efficiency in solving tasks of varying difficulty levels and domains (algebra, geometry, logic, and numbers), as well as different formats (text and image-based). Testing included tasks in both Serbian and English, allowing for the evaluation of language barriers in tool performance. The results indicate that tools perform better with text-based task formats, with o1-preview standing out for its exceptionally high accuracy in this format. All tools achieve the highest precision in numbers and algebra, while results are significantly lower in geometry and logic, highlighting challenges in processing visual information and logical reasoning. The conclusions of this study emphasize the importance of generative AI in improving mathematics education but highlight the need for further development of tools that can better handle visual tasks, support local languages, and be more specialized in solving mathematical problems in general.