Noometry

Model comparison

Gemma 2 9B vs Grok-2 (Dec 2024)

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.4 to 25.7 on the Noometry Index.

Last verified . 28 shared benchmarks.

Gemma 2 9B Google

25.7

Rank #334 Confirmed

Grok-2 (Dec 2024) xAI

33.4

Rank #233 Confirmed

Summary

  • They share 28 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and Grok-2 (Dec 2024) in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Grok-2 (Dec 2024) leads 29.8 to 9.7.
  • The biggest single-benchmark swing is MATH Level 5: 21% for Gemma 2 9B and 63.5% for Grok-2 (Dec 2024).
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9BGrok-2 (Dec 2024)
ProviderGooglexAI
Noometry Index25.733.4
Released2024-06-242024-08-13
WeightsOpenProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3534

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Grok-2 (Dec 2024) leads

Gemma 2 9B: 29.0 (#299), Grok-2 (Dec 2024): 32.9 (#254)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LiveBench Coding22.5%46.4%
LMArena Coding11741288
WeirdML—22.2%
BigCodeBench Instruct34.7%—
BigCodeBench Complete40.6%—

Reasoning Grok-2 (Dec 2024) leads

Gemma 2 9B: 15.3 (#306), Grok-2 (Dec 2024): 16.4 (#294)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LiveBench Reasoning15.2%54.8%
LMArena Hard Prompts11721272
LiveBench Data Analysis36.4%54.5%
Epoch Capabilities Index119.78130.48
LiveBench28.7%54.3%
SimpleBench—22.7%
DTBench—65.2%
PIQA83.7%—

Math Grok-2 (Dec 2024) leads

Gemma 2 9B: 9.3 (#312), Grok-2 (Dec 2024): 19.6 (#278)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
OTIS Mock AIME 2024-20250.6%11.5%
LiveBench Math19.8%54.9%
LMArena Math11831283
MATH Level 521%63.5%
FrontierMath (Feb 2025 set)—0.7%
GSM8K84.9%—

Knowledge Grok-2 (Dec 2024) leads

Gemma 2 9B: 9.7 (#300), Grok-2 (Dec 2024): 29.8 (#225)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
GPQA Diamond27.5%53.8%
LMArena Expert11471254
Confabulations—20.1%
BoolQ85.7%—
MMLU72.1%—

Multilingual Grok-2 (Dec 2024) leads

Gemma 2 9B: 36.8 (#233), Grok-2 (Dec 2024): 43.3 (#183)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LMArena Non-English11881282
LMArena Chinese11851289
LMArena French11891317
LMArena German11861287
LMArena Japanese11441244
LMArena Korean11371237
LMArena Russian12001286
LMArena Spanish12001281

Instruction Following Grok-2 (Dec 2024) leads

Gemma 2 9B: 57.8 (#264), Grok-2 (Dec 2024): 67.1 (#197)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LiveBench Instruction Following52.6%69.6%
LMArena Instruction Following11781270

Long Context Grok-2 (Dec 2024) leads

Gemma 2 9B: 36.3 (#228), Grok-2 (Dec 2024): 38.7 (#186)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LMArena Longer Query11971276

Writing & Preference Grok-2 (Dec 2024) leads

Gemma 2 9B: 32.3 (#276), Grok-2 (Dec 2024): 48.8 (#192)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LMArena Text12071305
LMArena Creative Writing12061284
LMArena Multi-Turn11931290
LiveBench Language25.5%45.6%
Short-Story Creative Writing—63.6%
EQ-Bench Creative Writing841—

Frequently asked questions

Is Gemma 2 9B better than Grok-2 (Dec 2024)?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.4 to 25.7 on the Noometry Index.

Is Gemma 2 9B or Grok-2 (Dec 2024) better for coding?

Grok-2 (Dec 2024) scores higher on coding benchmarks: 32.9 versus 29.0 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Grok-2 (Dec 2024) share?

28 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Grok-2 (Dec 2024) has 34.

Related comparisons

Go deeper