Evaluating large and small language models for programming problem solving.

Abstract

The transformation of natural language into code is evolving rapidly, driven by advances in Large and Small Language Models (LLMs and SLMs). Although it demonstrates great potential in code generation, the effectiveness of these models in real programming scenarios is still uncertain, especially considering different types of problem and levels of difficulty. This study evaluates the accuracy of Large Language Models (GPT-4, LLAMA 3, CLAUDE 3 SONNET and GEMINI PRO 1.0) on 100 LeetCode and BeeCrowd problems, in addition to investigating the performance of Small Language Models (LLAMA 3.2 3B, GEMMA 2 9B, PHI-4 14Band DEEPSEEK-R1 14B) on 280 Codeforces problems. The results show that, in the group of LLMs, GPT-4 led with 78 correct solutions, showing greater ease in lower-level problems. Among SLMs, PHI-4 14B stands out by solving 63% of problems, significantly outperforming other models, which apply rates lower than 23%. These results indicate the potential of LLMs and SLMs as settlement residents, but also highlight the significant variation in success rates depending on the complexity of the problems. Therefore, despite helping significantly, they should not be adopted as independent solutions. In the case of SLMs, although PHI-4 14B presents promising results, there are still limitations.