Why Arabic LLM evaluation is harder than English LLM evaluation
Evaluating an Arabic large language model in 2026 is a meaningfully different exercise than evaluating an English LLM, and the off-the-shelf evaluation playbooks transferred from English benchmarking miss the most important failure modes. Arabic is not one language but a constellation: Modern Standard Arabic (MSA, Fusha) for formal written communication; roughly 30 distinct major dialects (Khaleeji, Najdi, Hejazi, Egyptian, Levantine, Maghrebi, Iraqi, Sudanese, and many others) with varying mutual intelligibility; a Classical Arabic register for religious-and-literary contexts; and a heavily code-switched register where modern Saudi speech mixes Arabic and English in technical and commercial contexts.
A model that scores well on MSA benchmarks can fail badly on Saudi dialect comprehension, religious-and-cultural appropriateness, code-switched conversation, and Arabic retrieval-augmented generation over real Saudi enterprise documents. Evaluating an Arabic LLM seriously means evaluating it across all of these axes, with infrastructure that does not yet exist as an off-the-shelf bundle. This guide gives the seven evaluation axes, the available evaluation infrastructure as of 2026, a 6-to-8-week serious-evaluation methodology, and a candid view of the candidate-model landscape — Allam, Claude, GPT, Gemini, Llama-with-Arabic-tunes, Falcon, and Jais.
The stakes for getting Arabic LLM evaluation right are real and rising. The Saudi enterprise AI deployment market reached an estimated $1.2 to $1.8 billion in 2024 and is on a 35-to-45 percent CAGR; selecting the wrong model for a customer-facing Arabic deployment in regulated-industry use cases (banking, healthcare, government) can produce both reputational damage and regulatory consequences under the SDAIA AI ethics framework. The cost of a 6-to-8-week serious evaluation is small relative to the multi-million-dollar mistakes that off-the-shelf benchmarking enables.
The seven evaluation axes
Axis 1 — MSA fluency. Modern Standard Arabic comprehension and generation is the foundation. Test grammar, morphology (Arabic’s rich inflectional system), agreement (gender, number, definiteness), and stylistic register (formal-business, legal-formal, journalistic-formal). Models trained primarily on web Arabic frequently fail formal-register tests. Use ArabicMMLU, AraBench, and a hand-curated Saudi-business-Arabic corpus for this axis. A specific evaluation that catches many models off-guard: legal-Arabic register, where the morphology and lexical specificity of Saudi commercial-law and Sharia-court Arabic is far stricter than the web-Arabic register most models are trained on.
Axis 2 — Dialect handling. Test the model on Najdi, Hejazi, and Khaleeji (the Saudi-relevant dialects) plus Egyptian and Levantine (the most commonly encountered cross-Gulf dialects) for both comprehension and generation. Most models handle MSA-to-dialect translation reasonably but struggle with dialect-to-dialect translation and with generating dialect-appropriate responses. The dialect axis is where Allam, the Jais family, and Saudi-tuned Llama variants tend to outperform the global frontier models. The evaluation should distinguish between dialect comprehension (the model understands a Najdi-speaking user) and dialect generation (the model produces natural Najdi-register output) — the second is materially harder than the first and matters for consumer-facing deployments.
Axis 3 — Religious and cultural fluency. Saudi commercial and consumer deployments require appropriate handling of religious context (Islamic terminology, Quranic citations, Hadith references, religious-legal terminology) and cultural context (Saudi-specific protocols, royal-family references, regional sensitivities, Hajj-and-Umrah terminology). Failures on this axis produce material reputational and regulatory risk. Build a hand-curated test set of 200-500 prompts spanning religious-knowledge, cultural-appropriateness, and Saudi-specific protocol questions, and grade with native-speaker reviewers rather than automated metrics alone. The evaluation should specifically include adversarial prompts that probe the model’s behavior on religiously-sensitive topics, content involving the Royal Family, and culturally-specific scenarios where Western training-data norms may produce inappropriate outputs.
Axis 4 — Code-switching. Saudi technical and commercial conversation routinely code-switches between Arabic and English mid-sentence, with technical vocabulary frequently in English, social-and-greeting vocabulary in Arabic, and numeric-and-financial vocabulary in either. Models trained primarily on English-only or Arabic-only data handle code-switched input poorly. Test with a hand-curated Saudi-business-conversation corpus and grade for code-switched output naturalness as well as comprehension accuracy. The evaluation should also test Arabic-to-English-to-Arabic round-trip — code-switched input that produces a translation-flavored response rather than a naturally code-switched response is a sign that the model is internally normalizing the input rather than handling it natively.
Axis 5 — RAG over Arabic documents. Most enterprise deployments use retrieval-augmented generation over Arabic-language source documents (Saudi government regulations, Aramco technical documents, banking regulations from SAMA, healthcare records, legal opinions). Evaluate Arabic RAG as a system: Arabic embedding-model quality, Arabic chunk-and-retrieve accuracy, and the LLM’s ability to cite and reason over Arabic-language retrieved context. Arabic embeddings — particularly for dialect content and mixed MSA-and-dialect documents — are a frequent weak point. The evaluation should include numeric-extraction tasks over Arabic financial documents (Arabic numerals are sometimes Western, sometimes Eastern Arabic — ‘٠١٢٣’ versus ‘0123’ — and models trained primarily on Western-numeral data fail on Eastern-Arabic numerals).
Axis 6 — Instruction-following: Arabic vs English. Test the same set of complex instructions in both Arabic and English and compare adherence. A common failure mode is that models trained primarily on English instruction data follow English instructions more reliably than Arabic ones, even when Arabic comprehension is otherwise strong. The English-Arabic instruction-following gap is the most important “real deployment” indicator for Saudi enterprise applications. The evaluation should specifically test multi-step instructions, conditional logic in instructions, and instructions that require structured output (JSON, tables, formatted reports) — these are the patterns where the Arabic-instruction-following gap shows up most sharply.
Axis 7 — Hallucination and refusal calibration in Arabic. Test the model’s refusal patterns and hallucination rates on Arabic-language adversarial prompts, particularly in the religious-and-cultural-sensitivity domain. Models with strong English-language safety calibration frequently have weaker Arabic-language calibration, with both over-refusal (refusing benign Arabic queries) and under-refusal (engaging with adversarial Arabic queries that the English version would refuse) as failure modes. The evaluation should include cross-lingual jailbreak attempts — translating known English-language jailbreaks into Arabic and assessing whether the safety calibration generalizes — and direct Arabic-language adversarial prompts crafted by native speakers.
Available evaluation infrastructure
The 2026 Arabic LLM evaluation infrastructure is improving rapidly but remains less mature than English. Several resources are essential.
AraBench. A multi-task Arabic benchmark covering reading comprehension, sentiment, named-entity recognition, and translation. Useful as a baseline screen but does not capture the higher-order axes (cultural fluency, RAG, instruction-following).
Arabic MMLU. An Arabic-language adaptation of the MMLU benchmark spanning multiple knowledge domains. Useful for comparing models on Arabic-language general knowledge. Note that Arabic MMLU performance can be inflated by translation-from-English-MMLU artifacts; the most rigorous version uses native-Arabic-authored questions rather than translations.
ALUE (Arabic Language Understanding Evaluation). The Arabic equivalent of GLUE, covering several language-understanding tasks.
OALL (Open Arabic LLM Leaderboard). A public leaderboard run by HuggingFace with Arabic-specific evaluations; useful for tracking the candidate-model landscape over time. The OALL is best treated as a screen rather than a final-decision input — leaderboard results are subject to evaluation-set memorization in trained models and do not capture the higher-order axes.
AraGen and AraTrust. More recent evaluation suites focused on Arabic generation quality and Arabic safety/trust calibration.
Camel Arena and chatbot-arena Arabic. Live human-preference-based evaluation surfaces are emerging in 2025-2026 and are particularly valuable for the dialect-handling and instruction-following axes where automated metrics underperform.
In practice, none of these alone is sufficient for a serious enterprise evaluation. The right approach is to use the off-the-shelf benchmarks as a screening layer, then build hand-curated Saudi-specific evaluation sets for the higher-order axes. Budget for 200 to 500 hours of native-Arabic-speaker evaluator time, ideally split across MSA, Saudi dialect, religious-cultural, and Saudi-business contexts. The evaluator pool should include both younger digital-native Arabic speakers (who handle code-switching and modern dialect best) and older formal-register Arabic speakers (who handle MSA and Classical Arabic best); a single evaluator profile produces systematically biased results.
A 6-to-8-week serious-evaluation methodology
Week 1 — Scoping and corpus construction. Define the deployment context (industry, audience, language register), identify the candidate models, and begin construction of the Saudi-specific evaluation corpus. Recruit 3 to 5 native-Saudi evaluators with subject-matter relevance.
Week 2 — Baseline benchmarking. Run the off-the-shelf benchmarks (AraBench, ArabicMMLU, ALUE, OALL) on the candidate models. Use these as a screening layer; eliminate models that fail basic Arabic comprehension.
Week 3 — MSA-and-dialect testing. Run the hand-curated MSA and dialect test sets, with dual scoring (automated metric and native-speaker grading).
Week 4 — Religious-cultural-fluency testing. Run the religious-and-cultural test set with native-speaker grading, supplemented by review from a Saudi cultural-context advisor where the deployment has reputational exposure.
Week 5 — Code-switching and instruction-following. Run the code-switched conversation corpus and the Arabic-vs-English instruction-following comparison.
Week 6 — RAG and end-to-end testing. Build a representative Arabic RAG pipeline using the deployment-realistic embedding model, retrieval pipeline, and prompt structure. Run the RAG-specific test set.
Week 7 — Adversarial and safety testing. Run Arabic-language adversarial prompts including religious-context attacks, dialect-context attacks, and code-switched attacks. Document refusal-and-hallucination behavior.
Week 8 — Synthesis and decision. Aggregate the seven-axis scorecard, weight by the deployment context, and produce a recommendation. Document residual risks and ongoing-evaluation requirements.
Compress this timeline only at significant cost to the quality of the evaluation. A six-week serious evaluation is the minimum for an enterprise commitment. A common mistake is running the methodology once at procurement and then never re-running it — the candidate-model landscape changes substantially every 6 to 9 months, and a model selected in early 2024 may be substantially outperformed by a competitor in late 2024. Plan for periodic re-evaluation every 9 to 12 months.
The 2026 candidate-model landscape
Allam. The Saudi sovereign LLM developed under SDAIA leadership. Strongest on MSA-and-dialect Saudi content, religious-cultural fluency, and SDAIA-aligned safety calibration. Weaker on frontier reasoning, advanced coding, and English-language tasks compared to global frontier models. The right choice for sovereign-mandated workloads and where Saudi-cultural-context primacy outweighs frontier capability.
Claude (Anthropic). Strong general capability with rapidly improving Arabic. Particularly strong instruction-following in Arabic and strong religious-cultural calibration. The default choice for many Saudi enterprise deployments where frontier capability is required and the deployment is not sovereign-mandated. Claude’s Arabic capability has improved materially across the 4.x generation; evaluations from 2023 are no longer reliable indicators.
GPT-4 family (OpenAI). Strong general capability with mature Arabic. Particularly strong on code-switching and on Arabic-English bilingual tasks. Slightly less religious-cultural calibration than Claude in 2026 but improving.
Gemini (Google). Strong general capability with rapidly improving Arabic. Particularly strong on multimodal Arabic tasks (Arabic OCR, Arabic-document understanding). A leading choice for document-heavy Arabic workloads.
Llama with Arabic tunes. A growing ecosystem of Llama-derivative models with Arabic post-training. Variable quality; the better entries (Jais-Llama, ALLaM-Llama) are competitive on MSA and Saudi dialect but lag the frontier on reasoning and instruction-following. The right choice for cost-sensitive deployments and self-hosted sovereign requirements.
Falcon and Jais. The TII-anchored Falcon family and the G42-anchored Jais family are the strongest Gulf-developed open Arabic models. Particularly strong on Arabic-first training-data composition and dialect handling. Weaker on frontier capability than the closed leaders. The right choice for Gulf-sovereign deployments where open weights and Arabic-first training are priorities.
A seventh emerging category worth tracking: small specialized Arabic models (1-7B parameters) tuned for specific verticals (legal, medical, financial). These models are not frontier-capable but materially outperform larger general models on their specific verticals and have substantially lower deployment costs.
The decision across these candidates is rarely a single-model choice; the dominant 2026 enterprise pattern is multi-model with Allam or Jais for sovereign-mandated workloads, Claude or GPT for frontier-capability workloads, and Gemini for multimodal Arabic workloads. Architect for model substitutability rather than a single-model commitment. The architectural pattern that pays dividends is a model-router layer that routes specific prompts to specific models based on workload classification, with consistent observability and evaluation across the routed traffic — this lets the deployment evolve with the candidate-model landscape rather than locking in a single-model commitment.
Production telemetry as continuous evaluation
The 6-to-8-week serious-evaluation methodology produces the procurement decision; the production-deployment phase requires a complementary discipline of continuous evaluation through production telemetry. The pattern that works in 2026 enterprise deployments is to instrument the deployment with structured logging of input language register (MSA, dialect, code-switched, English), output quality signals (user-feedback ratings, completion rates, escalation-to-human rates), and adversarial-prompt detection. The telemetry feeds a quarterly re-evaluation that checks whether the model is still performing as expected on the seven axes, whether new failure modes have emerged from production traffic patterns that the original evaluation did not anticipate, and whether new candidate models in the rapidly-evolving landscape would outperform the deployed model on the production traffic distribution. The discipline pays dividends specifically because the Arabic LLM landscape is moving faster than equivalent English deployments — quarterly re-evaluation catches model improvements that 18-month procurement cycles miss. A specific failure mode that production telemetry catches early: distribution drift in which the actual user population uses dialect, code-switched, or domain-specific language at frequencies different from the original evaluation corpus, exposing model weaknesses that the procurement evaluation did not surface. The teams that build this continuous-evaluation discipline outperform peers on Arabic-deployment outcomes by a meaningful margin and avoid the multi-million-dollar redeployments that follow major model failures discovered in production rather than evaluation.
For deeper reading: How to read SDAIA strategy, How to track the Humain roadmap, Players: Allam, How to pick a Saudi cloud provider.