Performance of an AI tool in solving non-standard mathematics competition problems

Abstract

For some time now, researchers around the world have been examining the effects of using AI in mathematics education to provide additional support and assistance to students. One line of research focuses on helping students who wish to participate in math competitions by solving more complex mathematical problems. In addition to regular national math competitions, which allow students to progress to international mathematical Olympiads, there are competitions aimed at popularizing mathematics and developing logical thinking in students. One such competition is the international Kangaroo competition. In this paper, we analyze the performance of the AI Math Solver on the Interactive Mathematics platform in solving problems from the 2024 Kangaroo competition for students in the 3rd and 4th grades of elementary school, as well as the 7th and 8th grades of elementary school, and the 3rd and 4th grades of high school. The tasks were uploaded in the form of images (screenshots), both in Serbian and English, because in the formulation of the tasks and/or provided answers for the Kangaroo competition, images often appear. Out of a total of 84 tasks, both in Serbian and in English, it correctly solved 24, which is just under 30% success in both cases. Furthermore, some tasks solved in Serbian were not solved in English, and vice versa. Additionally, differences were found in the distribution of correct answers among tasks of different difficulty levels.