Noometry

Model comparison

Gemma 2 9B vs Mistral Large

Mistral Large is the stronger model overall, scoring 31.9 to 25.9 on the Noometry Index.

Last verified . 32 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

Mistral Large Mistral AI

31.9

Rank #263 Confirmed

Summary

  • They share 32 benchmarks with published results for both. Gemma 2 9B scores higher in 1 category and Mistral Large in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Mistral Large leads 30.1 to 9.7.
  • The biggest single-benchmark swing is MATH Level 5: 21% for Gemma 2 9B and 50.3% for Mistral Large.

Side by side

Gemma 2 9BMistral Large
ProviderGoogleMistral AI
Noometry Index25.931.9
Released2024-06-242024-02-26
WeightsOpenOpen
Context window—131K
Max output—16K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked3551

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Large leads

Gemma 2 9B: 29.4 (#304), Mistral Large: 34.3 (#240)

BenchmarkGemma 2 9BMistral Large
BigCodeBench Instruct34.7%30%
LiveBench Coding22.5%47.1%
LMArena Coding11731277
BigCodeBench Complete40.6%38.3%
SciCode—36.2%
ALE-Bench—264.7
HumanEval+—62.2%
MBPP+—59.5%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, Mistral Large: 28.6 (#89)

BenchmarkGemma 2 9BMistral Large
Berkeley Function Calling Leaderboard—38.4%

Reasoning Too close to call

Gemma 2 9B: 15.9 (#309), Mistral Large: 15.8 (#310)

BenchmarkGemma 2 9BMistral Large
LiveBench Reasoning15.2%43.5%
LMArena Hard Prompts11711257
LiveBench Data Analysis36.4%50.1%
Epoch Capabilities Index119.83128.52
LiveBench28.7%48.4%
SimpleBench—22.5%
CritPt—0%
DTBench—65.1%
LMCA—16.7%
ForecastBench—57.1
PIQA83.7%—

Math Mistral Large leads

Gemma 2 9B: 9.9 (#318), Mistral Large: 18.2 (#291)

BenchmarkGemma 2 9BMistral Large
OTIS Mock AIME 2024-20250.6%8.5%
LiveBench Math19.8%42.5%
LMArena Math11831262
MATH Level 521%50.3%
Omni-MATH—28.1%
FrontierMath (Feb 2025 set)—0.3%
GSM8K84.9%—

Knowledge Mistral Large leads

Gemma 2 9B: 9.7 (#305), Mistral Large: 30.1 (#230)

BenchmarkGemma 2 9BMistral Large
GPQA Diamond27.5%51.3%
LMArena Expert11471232
MMLU72.1%80%
MMLU-Pro—59.9%
Confabulations—21.4%
Vectara Hallucination Rate—4.5%
GPQA (HELM)—43.5%
BoolQ85.7%—

Multilingual Mistral Large leads

Gemma 2 9B: 36.6 (#238), Mistral Large: 40.0 (#219)

BenchmarkGemma 2 9BMistral Large
LMArena Non-English11881237
LMArena Chinese11851240
LMArena French11901325
LMArena German11861254
LMArena Japanese11441188
LMArena Korean11371202
LMArena Russian12001257
LMArena Spanish12001268

Instruction Following Mistral Large leads

Gemma 2 9B: 57.6 (#269), Mistral Large: 67.9 (#191)

BenchmarkGemma 2 9BMistral Large
LiveBench Instruction Following52.6%67.9%
LMArena Instruction Following11781249
IFEval—87.7%

Long Context Mistral Large leads

Gemma 2 9B: 36.3 (#233), Mistral Large: 38.3 (#199)

BenchmarkGemma 2 9BMistral Large
LMArena Longer Query11971261

Writing & Preference Mistral Large leads

Gemma 2 9B: 32.1 (#281), Mistral Large: 40.7 (#242)

BenchmarkGemma 2 9BMistral Large
LMArena Text12071266
LMArena Creative Writing12061243
EQ-Bench Creative Writing841985
LMArena Multi-Turn11931260
LiveBench Language25.5%39.4%
Short-Story Creative Writing—69%
WildBench—80.1%

Frequently asked questions

Is Gemma 2 9B better than Mistral Large?

Mistral Large is the stronger model overall, scoring 31.9 to 25.9 on the Noometry Index.

Is Gemma 2 9B or Mistral Large better for coding?

Mistral Large scores higher on coding benchmarks: 34.3 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Mistral Large share?

32 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Mistral Large has 51.

Related comparisons

Go deeper