Georgian Language Benchmark: Two Biggest AI Labs Clash in a Language That Doesn't Play by Anyone's Rules

A developer has launched a comprehensive benchmark to test how top AI models handle the Georgian language, a unique, non-Indo-European language with a complex grammatical structure and its own alphabet. The project, submitted for a Kaggle challenge, evaluates models on five specific tasks: number conversion, verb conjugation, translation, cloze tests, and grammatical correction. Despite facing significant constraints—including a limited daily budget on the Kaggle platform and a personal injury—the author manually curated datasets to ensure high-quality testing. The findings reveal that while modern AI models have made progress in understanding Georgian, they still struggle with accurate production, particularly with complex number systems and nuanced grammar. The benchmark highlights the challenges of evaluating LLMs in low-resource or linguistically unique languages, emphasizing that 'cheap' models can sometimes outperform larger ones depending on the specific task requirements.