Noometry

Model comparison

Claude 3.5 Haiku vs Mistral Large

Mistral Large is the stronger model overall, scoring 31.6 to 28.9 on the Noometry Index.

Last verified . 43 shared benchmarks.

Claude 3.5 Haiku Anthropic

28.9

Rank #309 Confirmed

Mistral Large Mistral AI

31.6

Rank #258 Confirmed

Summary

  • They share 43 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 3 categories and Mistral Large in 6 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Mistral Large leads 30.0 to 18.7.
  • The biggest single-benchmark swing is BigCodeBench Complete: 59% for Claude 3.5 Haiku and 38.3% for Mistral Large.
  • Mistral Large has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuMistral Large
ProviderAnthropicMistral AI
Noometry Index28.931.6
Released2024-10-222024-02-26
WeightsProprietaryOpen
Context window—131K
Max output—16K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked4951

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Large leads

Claude 3.5 Haiku: 32.6 (#258), Mistral Large: 33.8 (#240)

BenchmarkClaude 3.5 HaikuMistral Large
SciCode27.4%36.2%
BigCodeBench Instruct46.1%30%
LiveBench Coding51.4%47.1%
LMArena Coding12871277
BigCodeBench Complete59%38.3%
Aider Polyglot28%—
WeirdML30.7%—
CadEval32%—
ALE-Bench—264.7
HumanEval+—62.2%
MBPP+—59.5%

Agentic & Tool Use Too close to call

Claude 3.5 Haiku: 27.7 (#90), Mistral Large: 28.4 (#83)

BenchmarkClaude 3.5 HaikuMistral Large
Berkeley Function Calling Leaderboard—38.4%
BALROG19.3%—

Reasoning Claude 3.5 Haiku leads

Claude 3.5 Haiku: 17.1 (#286), Mistral Large: 15.4 (#305)

BenchmarkClaude 3.5 HaikuMistral Large
CritPt0%0%
LiveBench Reasoning28.1%43.5%
LMArena Hard Prompts12511257
DTBench56.7%65.1%
LiveBench Data Analysis48.5%50.1%
Epoch Capabilities Index127.15128.52
LiveBench43.5%48.4%
SimpleBench—22.5%
LMCA—16.7%
ForecastBench—57.1

Math Mistral Large leads

Claude 3.5 Haiku: 13.7 (#294), Mistral Large: 17.1 (#285)

BenchmarkClaude 3.5 HaikuMistral Large
OTIS Mock AIME 2024-20254.3%8.5%
Omni-MATH22.4%28.1%
LiveBench Math35.5%42.5%
LMArena Math12441262
MATH Level 546.4%50.3%
FrontierMath (Feb 2025 set)0.3%0.3%

Knowledge Mistral Large leads

Claude 3.5 Haiku: 18.7 (#276), Mistral Large: 30.0 (#224)

BenchmarkClaude 3.5 HaikuMistral Large
GPQA Diamond38.1%51.3%
MMLU-Pro60.5%59.9%
Confabulations36.7%21.4%
GPQA (HELM)36.3%43.5%
LMArena Expert12081232
MMLU74.3%80%
Vectara Hallucination Rate—4.5%

Multimodal Not comparable

Claude 3.5 Haiku: 26.3 (#113), Mistral Large: —

BenchmarkClaude 3.5 HaikuMistral Large
LMArena Vision1092—
GeoBench34%—

Multilingual Too close to call

Claude 3.5 Haiku: 40.2 (#213), Mistral Large: 40.1 (#214)

BenchmarkClaude 3.5 HaikuMistral Large
LMArena Non-English12381237
LMArena Chinese12291240
LMArena French12621324
LMArena German12371254
LMArena Japanese11751187
LMArena Korean11721202
LMArena Russian12531257
LMArena Spanish12611268

Instruction Following Mistral Large leads

Claude 3.5 Haiku: 63.1 (#229), Mistral Large: 68.1 (#186)

BenchmarkClaude 3.5 HaikuMistral Large
LiveBench Instruction Following61.9%67.9%
IFEval79.2%87.7%
LMArena Instruction Following12411249

Long Context Too close to call

Claude 3.5 Haiku: 38.3 (#195), Mistral Large: 38.3 (#194)

BenchmarkClaude 3.5 HaikuMistral Large
LMArena Longer Query12611262

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.9 (#228), Mistral Large: 40.9 (#237)

BenchmarkClaude 3.5 HaikuMistral Large
LMArena Text12551266
LMArena Creative Writing12331243
Short-Story Creative Writing73.5%69%
EQ-Bench Creative Writing1146985
WildBench76%80.1%
LMArena Multi-Turn12651260
LiveBench Language35.4%39.4%

Frequently asked questions

Is Claude 3.5 Haiku better than Mistral Large?

Mistral Large is the stronger model overall, scoring 31.6 to 28.9 on the Noometry Index.

Is Claude 3.5 Haiku or Mistral Large better for coding?

Mistral Large scores higher on coding benchmarks: 33.8 versus 32.6 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Mistral Large share?

43 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Mistral Large has 51.

Related comparisons

Go deeper