HateBench: A benchmark for models against Hindi Hateful Speech

Overview

HateBench represents the most comprehensive evaluation of Large Language Models on Hindi hate speech detection to date. Over a two-week period, I benchmarked 67 models from 23 providers on a curated dataset of 150 Hindi social media posts, some collected from Instagram comments and others sourced from the Indo-HateSpeech dataset, measuring not just accuracy but cost efficiency and latency, metrics that matter for production deployment.

Dataset Attribution

The Indo-HateSpeech dataset (Kaware, 2024) is a Hindi-English code-mixed dataset designed for identifying hate speech on social media platforms. Given the multilingual nature of Indian social media users, code-mixing—the blending of Hindi and English—is prevalent and poses unique challenges for content moderation.

The Bottom Line

Critical Implications:

• 74.7% accuracy means roughly 1 in 4 classifications are still wrong—unacceptable for safety-critical moderation
• 30.4% false positive rate threatens free speech by over-flagging political/religious commentary
• 20-40% accuracy on pure Hindi script disadvantages rural and formal Hindi speakers
• 52.9% of positive or non-harmful content with emojis is misclassified as hate; models don't understand Indian social media tone
• Cost varies 423× for 0.7% accuracy difference—market inefficiency creates exploitable arbitrage, and newer models haven't fixed this gap

What We Tested

The benchmark evaluates models on a binary classification task (with models technically outputting 3 classes):

• Hate Speech: Toxic language, slurs, threats, dehumanization, religious attacks, gendered harassment (75 examples, 50%)
• Positive / Non-Harmful: Political commentary, religious expressions, banter, patriotism, gratitude, and supportive language (75 examples, 50%)
• Neutral Output: Not part of the benchmark labels, yet all 67 models still predicted it 9-27 times, revealing systematic confusion on the non-hate boundary

Content distribution: 44.7% contain emojis, 84% Romanized Hindi, 6.7% pure Devanagari, 4% mixed script. Models were tested on culturally-specific Indian social media content from Instagram comments and the Indo-HateSpeech dataset.

Evaluation Prompt

The following system prompt was sent to every model before each classification. This ensures the results are directly comparable across providers.

Top Performers & Cost Analysis

Cost per 150 examples and per correct prediction shown. Trinity and Sarvam were free through OpenRouter/Sarvam APIs during benchmark. Pricing subject to change. Lower cost/correct = better value.

Notable Underperformers

• Molmo 2 8B: 42.7% accuracy — the lowest of all 67 models. A vision-native model that completely fails on text-only moderation.
• Phi-4: 47.3% accuracy with 66.7% false positive rate on political content and 0% internet slang detection.
• Mercury 2: 48.7% accuracy — Inception's model is near the bottom despite being a general-purpose contender.
• Hunyuan A13B: 50.7% accuracy — Tencent's smaller model is well below the 62.7% average.
• Llama 4 Maverick: 51.3% accuracy with 56.0% false positive rate on non-harmful content, making it the worst offender for over-flagging.
• Mistral Medium 3.5: 56.0% accuracy — Mistral's mid-tier model underperforms every Google model at 5× the cost.
• Qwen3 235B: Massive 235B parameter model achieves only 57.3% accuracy — below the 62.7% average. Size still doesn't scale.

New Models Since the Original Post

The expanded run added 20 more models, including several that changed the leaderboard. Google's Gemini 3.1 Flash Lite is the standout, it jumps past Trinity Large to become the first model to crack 74% on this benchmark, and it does so at 1.1s latency for two cents. Anthropic's Claude Opus 4.8 matches the 70.7% tier but at $0.58 per run, while GPT-5.2 Pro also hits 70.7% at a staggering $3.11. Gemma 4 31B slots neatly into the 70% club at $0.007, confirming Google's architecture is the most consistent for this task. On the flip side, vision-native models like Molmo 2 8B collapse entirely on text-only moderation, and Meta's Llama 4 Maverick remains a false-positive machine.

None of the new releases fundamentally solve the core problems: Devanagari blindness, emoji misinterpretation, and the 30%+ false-positive rate on political and religious content. They just raise the ceiling slightly.

Dataset Deep Dive

Before diving into model performance, it's crucial to understand what we tested against. The dataset composition reveals critical imbalances:

• 84% Romanized Hindi (Hinglish): Mixed Hindi-English script that dominates Indian social media
• 6.7% Pure Devanagari: Traditional Hindi script that models struggle with dramatically
• 4.0% Mixed script: Code-switched content combining both scripts
• 44.7% contain emojis: Critical context markers that models misinterpret

Critical Insights

1. The Accuracy Ceiling: 71.3% is the Hard Limit

Hindi hate speech detection maxes out at 74.7% accuracy (Gemini 3.1 Flash Lite). The best models cluster at 70-74%, suggesting a soft ceiling for current architectures. Average across all 67 models: 62.7%.

The Devanagari Problem: Models achieve 100% accuracy on Romanized Hindi but only 20-40% on pure Devanagari script. This isn't semantic understanding—it's transliteration dependency. Models translate Hindi → English → classify rather than understanding Hindi directly.

2. Cost-Performance Disconnect: Price ≠ Quality

The cost differences are staggering for marginal gains:

The ROI paradox: Gemma 3n E4B is 423× more cost-efficient than GPT-5.2 Pro despite nearly identical accuracy (70.0% vs 70.7%). Google models average 5× cheaper than OpenAI with 2.3% higher accuracy. The new Gemini 3.1 Flash Lite breaks the ceiling at 74.7% for just $0.02 — a genuine leap in the speed-accuracy-cost triangle.

3. The Script Barrier: Devanagari Blindness

All models exhibit catastrophic performance drops on pure Hindi script:

• Claude Opus 4.6: 100% on Hinglish → 20% on Devanagari (80% drop)
• Gemini 3 Flash: 100% on Hinglish → 40% on Devanagari (60% drop)
• Gemini 3 Pro: 100% on Hinglish → 20% on Devanagari (80% drop)

This reveals architectural bias toward Latin-script training data. Rural users, formal contexts, and regional Hindi speakers are systematically disadvantaged.

4. False Positive Crisis: 30.4% Over-Flagging

Models flag nearly a third of positive or non-harmful content as hate speech, with systematic biases:

• Religious content: 34.0% false positive rate
• Political commentary: 33.3% to 66.7% FP rate depending on model
• Worst offenders: Llama 4 Maverick (56.0%), Ministral 14B (49.3%), Gemma 3 27B (45.3%)

5. Emoji Context Collapse: 😂 Triggers False Positives

Laughing emojis cause catastrophic misclassification. Models interpret them as mockery markers rather than tone softeners:

• Positive / non-harmful + 😂 false positive rate: 52.9% (Claude Opus, Gemini Flash, Gemma 27B)
• Worst offender: Grok 4.1 Fast at 58.8% FP rate
• The paradox: Models over-index on 😂 as aggression when paired with criticism, but under-index when paired with actual slurs

Real-world failure: "BJP+400🚩🚩❤❤❤😈😈" (positive partisan slogan) produced a fragmented split across hate, positive, and neutral outputs instead of being recognized as clearly non-harmful.

6. Political Figures as Trip Wires

Mentioning prominent Indian political figures triggers hate classification even in non-harmful contexts:

• Phi-4 & Ministral 14B: 66.7% FP rate on non-harmful political posts
• Gemma 3 27B: 50.0% FP rate
• Average across models: ~40%

Root cause: Training data includes Western political hate speech mapped onto Indian figures, but Indian political discourse has different toxicity norms. Criticism ≠ hate in Indian culture, but models don't distinguish this.

7. The Parameter Paradox: Size Doesn't Scale

Model size shows zero correlation with accuracy. Larger models often dramatically underperform smaller ones:

Only Gemini 3.1 Flash Lite and Trinity Large (est. 400B+) benefit from scale at the top end. But Flash Lite is tiny and fast, proving that efficiency and capability aren't in tension. For production, deploy Gemma 3n E4B or Gemini 3.1 Flash Lite — save 99% on compute with better results than models 50× their size.

8. Indic Models Compete at Zero Cost

Sarvam models—specifically trained on Indic languages—deliver competitive results free:

• Sarvam-105B: 60.7% accuracy at $0.00 (free tier)
• Sarvam-30B: 58.7% accuracy at $0.00
• Gap vs. Western models: Only 0.8-2.8 points behind at zero cost

Critical vulnerability: Sarvam models detect 0% of abbreviated internet slurs while Western models detect 75-100%. A hybrid approach (Sarvam first pass + GPT for slang) could deliver 65%+ accuracy at near-zero cost.

9. Internet Slang Detection Gap: Binary Failure

Abbreviated internet slurs show bimodal detection—some models perfect (100%), others completely blind (0%), with no middle ground:

• Perfect detection (8 models): Gemini 3 Flash/Pro, GPT-5.2/5.2 Pro/5.4 Pro, GLM-4.7, Grok 4.1 Fast, Mimo V2 Pro
• Complete failure (5 models): Sarvam-30B, Sarvam-105B, Phi-4, MiniMax M2.1, Mercury 2
• Attack vector: Bad actors can evade detection by using Indic-specific moderation plus abbreviated internet slurs

10. Confidence Calibration Crisis: Scores Are Unreliable

Models exhibit dangerous over-confidence, making confidence scores unusable for human triage:

• High-confidence errors: 916 predictions with >90% confidence that were WRONG
• Low-confidence successes: Only 98 predictions with <60% confidence that were correct
• 100% confidence errors: Gemini 3 Pro, Gemma 3 12B, Sarvam-105B all made mistakes with absolute certainty

11. Content Type Performance Gaps: Gendered Hate Blind Spot

Models excel at political/religious hate (100% detection) but systematically miss gendered harassment:

• Political hate: 100% detection (Claude Opus)
• Religious hate: 100% detection
• Female-directed abuse: Only 80% detection, with a 20% miss rate on slurs targeting women

Critical blind spot: Gendered and sexualized hate speech slips through 5× more often than political hate, despite being equally harmful. Training data bias toward public discourse over private harassment contexts.

12. Code-Switching Blind Spots

Models handle Hindi-English code-switching (71% accuracy) better than pure Hindi (29-57% accuracy), but struggle with language boundaries:

• Mixed Hindi-English: 71% average accuracy
• Pure Devanagari: 29-57% accuracy
• Pure English: 60-80% accuracy

Real-world impact: Rural/regional users using pure Hindi script, regional language mixes (Marathi + Hindi + English), and professional contexts using Devanagari are all disadvantaged.

13. Model Consensus Uncertainty: 17% Requires Human Review

26 examples (17.3% of dataset) show <60% model agreement. Even aggregated predictions are unreliable for culturally-complex content:

• Most contentious: "BJP+400🚩🚩❤❤❤😈😈" (positive partisan slogan) → 45% neutral, 30% hate, 25% positive split
• Perfect consensus wrong: 2 examples with 100% model agreement were both wrong
• Irreducible floor: 17% of content will always require human review regardless of model sophistication

Production Recommendations

Model Selection Strategy

Scenario	Recommended Model	Why
Best Overall	Gemini 3.1 Flash Lite	74.7% accuracy, $0.02/test, 1.1s latency — best speed-accuracy-cost triangle
Best Budget	Gemma 3n E4B	70.0% accuracy, $0.0011/test (297× cheaper), 1.49s latency
Best Premium	Trinity Large	71.3% accuracy, $0.00 (free tier) — if you can handle rate limits
Hybrid Strategy	Sarvam-105B + GPT-5.2	First pass free (60.7%), slang catch on uncertain cases → ~68% accuracy at <$0.05/1K examples
Strong Alternative	Hy3 Preview	70.7% accuracy at $0.0762/test — Tencent's dark horse for moderation

Why This Matters

With 600+ million Hindi speakers worldwide and explosive growth in Indian social media usage, effective content moderation for Indic languages isn't optional—it's essential. Yet current solutions systematically fail this market.

The Real-World Impact

• Platform Safety: 20% of gendered hate speech slips through while 30% of political commentary is over-flagged. This creates both under-moderation (harassment goes unchecked) and over-moderation (legitimate speech suppressed).
• Digital Divide: Models perform 50-80% worse on pure Devanagari script, systematically disadvantaging rural users, older demographics, and formal/professional contexts. This is algorithmic bias against non-English-script users.
• Cultural Blindness: Models trained on Western data flag Indian political discourse as hate 40-67% of the time. What constitutes criticism vs. hate differs radically between cultures—models don't know this.
• Economic Exploitation: Bad actors can bypass detection by using abbreviated internet slurs against Indic models (0% detection) or mixing abuse with Devanagari to confuse transliteration layers.

The Technical Debt

• Confidence scores are broken: 916 high-confidence errors vs 98 low-confidence successes means human review triage is impossible without reviewing 60%+ of content.
• Emojis are misread: 😂 triggers hate classification 52.9% of the time in positive or non-harmful content; models fundamentally misunderstand Indian social media tone.
• Consensus is unreliable: Even aggregating 67 model outputs, 17% of content has <60% agreement. Some edge cases will always require human judgment.

What HateBench Provides

HateBench delivers the first empirical foundation for making informed moderation decisions in the world's second-largest language market. With 67 models tested, critical insights identified, and cost/performance/latency metrics for each, this is the most comprehensive Hindi hate speech benchmark to date.

The path forward: Hybrid human-AI moderation with culturally-aware models, script-aware handling, and consensus-based routing. 71.3% accuracy isn't good enough; we need 90%+ for safety-critical applications. This benchmark shows us exactly where the gaps are.

Methodology & Limitations

Analysis based on: 67 models from 23 providers (Anthropic, OpenAI, Google, Sarvam AI, Meta, Mistral, Tencent, DeepSeek, Nvidia, Xiaomi, etc.) tested on 150 examples from the Indo-HateSpeech dataset. Metrics include accuracy, precision, recall, F1, confusion matrices, cost (USD), and latency. Results current as of June 2026.

Limitations: Small dataset (150 examples), culturally ambiguous edge cases, free-tier models may have rate-limiting affecting scores, latency measurements include API overhead not just inference time. Newer models were added incrementally; some results may reflect different API availability windows.