Noometry

Model comparison

Claude 3 Haiku vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.1 to 25.7 on the Noometry Index.

Last verified . 27 shared benchmarks.

Claude 3 Haiku Anthropic

25.7

Rank #333 Confirmed

Mistral Small Mistral AI

33.1

Rank #238 Confirmed

Summary

  • They share 27 benchmarks with published results for both. Claude 3 Haiku scores higher in 0 categories and Mistral Small in 9 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.6 to 29.8.
  • The biggest single-benchmark swing is MATH Level 5: 14.9% for Claude 3 Haiku and 46.8% for Mistral Small.
  • Mistral Small has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuMistral Small
ProviderAnthropicMistral AI
Noometry Index25.733.1
Released2024-03-072024-02-26
WeightsProprietaryOpen
Context window—256K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked3739

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small leads

Claude 3 Haiku: 26.1 (#320), Mistral Small: 33.5 (#242)

BenchmarkClaude 3 HaikuMistral Small
BigCodeBench Instruct39.4%36.1%
LMArena Coding11991363
BigCodeBench Complete50.1%46.6%
SciCode—26.5%
WeirdML9.8%—
LiveBench Coding—36.2%
CadEval12%—
ALE-Bench—497.62
HumanEval+68.9%—
MBPP+68.8%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, Mistral Small: 27.9 (#87)

BenchmarkClaude 3 HaikuMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Mistral Small leads

Claude 3 Haiku: 15.7 (#303), Mistral Small: 19.3 (#238)

BenchmarkClaude 3 HaikuMistral Small
Kagi LLM Benchmark34.2%37.8%
LMArena Hard Prompts11741336
DTBench50.1%70.9%
LMCA8.8%20.6%
CritPt—0%
LiveBench Reasoning—44.8%
LiveBench Data Analysis—53.7%
Epoch Capabilities Index118.3—
ForecastBench53.2—
LiveBench—44%
WinoGrande74.2%—

Math Mistral Small leads

Claude 3 Haiku: 9.2 (#313), Mistral Small: 15.4 (#287)

BenchmarkClaude 3 HaikuMistral Small
OTIS Mock AIME 2024-20251.8%5.8%
LMArena Math11881341
MATH Level 514.9%46.8%
LiveBench Math—39.9%

Knowledge Mistral Small leads

Claude 3 Haiku: 17.3 (#280), Mistral Small: 30.9 (#214)

BenchmarkClaude 3 HaikuMistral Small
GPQA Diamond36.3%47.5%
LMArena Expert11481293
MMLU73.8%68.7%
Confabulations34.2%—
Vectara Hallucination Rate—5.1%

Multimodal Mistral Small leads

Claude 3 Haiku: 23.1 (#123), Mistral Small: 32.9 (#91)

BenchmarkClaude 3 HaikuMistral Small
LMArena Vision9501142
ScienceQA72%—

Multilingual Mistral Small leads

Claude 3 Haiku: 36.1 (#238), Mistral Small: 45.7 (#164)

BenchmarkClaude 3 HaikuMistral Small
LMArena Non-English11781315
LMArena Chinese11551342
LMArena French11941337
LMArena German11741341
LMArena Japanese11021275
LMArena Korean11081260
LMArena Russian12041324
LMArena Spanish11651346

Instruction Following Mistral Small leads

Claude 3 Haiku: 61.5 (#242), Mistral Small: 66.6 (#204)

BenchmarkClaude 3 HaikuMistral Small
LMArena Instruction Following11741310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Claude 3 Haiku: 36.1 (#232), Mistral Small: 40.4 (#151)

BenchmarkClaude 3 HaikuMistral Small
LMArena Longer Query11911327

Writing & Preference Mistral Small leads

Claude 3 Haiku: 29.8 (#286), Mistral Small: 52.6 (#165)

BenchmarkClaude 3 HaikuMistral Small
LMArena Text11951338
LMArena Creative Writing11571304
LMArena Multi-Turn11901344
EQ-Bench Creative Writing717—
LiveBench Language—30.5%

Frequently asked questions

Is Claude 3 Haiku better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.1 to 25.7 on the Noometry Index.

Is Claude 3 Haiku or Mistral Small better for coding?

Mistral Small scores higher on coding benchmarks: 33.5 versus 26.1 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Mistral Small share?

27 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper