Noometry

Model comparison

Grok-2 (Dec 2024) vs o3-mini

o3-mini is the stronger model overall, scoring 36.7 to 33.4 on the Noometry Index.

Last verified . 34 shared benchmarks.

Grok-2 (Dec 2024) xAI

33.4

Rank #233 Confirmed

o3-mini OpenAI

36.7

Rank #205 Confirmed

Summary

  • They share 34 benchmarks with published results for both. Grok-2 (Dec 2024) scores higher in 2 categories and o3-mini in 6 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where o3-mini leads 38.3 to 29.8.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 11.5% for Grok-2 (Dec 2024) and 76.9% for o3-mini.

Side by side

Grok-2 (Dec 2024)o3-mini
ProviderxAIOpenAI
Noometry Index33.436.7
Released2024-08-132024-12-20
WeightsProprietaryProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$1.10
Output $ / M tokens—$4.40
Results tracked3451

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o3-mini leads

Grok-2 (Dec 2024): 32.9 (#254), o3-mini: 41.4 (#116)

BenchmarkGrok-2 (Dec 2024)o3-mini
WeirdML22.2%43.7%
LiveBench Coding46.4%82.7%
LMArena Coding12881378
Aider Polyglot—60.4%
SciCode—39.8%
GSO—1.3%
CadEval—54%

Agentic & Tool Use Not comparable

Grok-2 (Dec 2024): —, o3-mini: 29.3 (#79)

BenchmarkGrok-2 (Dec 2024)o3-mini
Cybench—22.5%

Reasoning Too close to call

Grok-2 (Dec 2024): 16.4 (#294), o3-mini: 16.3 (#295)

BenchmarkGrok-2 (Dec 2024)o3-mini
SimpleBench22.7%22.8%
LiveBench Reasoning54.8%89.6%
LMArena Hard Prompts12721366
DTBench65.2%68.8%
LiveBench Data Analysis54.5%70.6%
Epoch Capabilities Index130.48140.35
LiveBench54.3%75.9%
ARC-AGI-2—3%
ARC-AGI-1—34.5%
CritPt—0.3%
Chess Puzzles—17%
Mystery Game Puzzles—7%
LMCA—19%
ForecastBench—59.6

Math o3-mini leads

Grok-2 (Dec 2024): 19.6 (#278), o3-mini: 27.6 (#236)

BenchmarkGrok-2 (Dec 2024)o3-mini
OTIS Mock AIME 2024-202511.5%76.9%
LiveBench Math54.9%77.3%
LMArena Math12831396
MATH Level 563.5%96.5%
FrontierMath (Feb 2025 set)0.7%12.4%
FrontierMath (Tiers 1-3)—18.6%
FrontierMath Tier 4—0%
FrontierMath Tier 4 (v1)—4.2%

Knowledge o3-mini leads

Grok-2 (Dec 2024): 29.8 (#225), o3-mini: 38.3 (#135)

BenchmarkGrok-2 (Dec 2024)o3-mini
GPQA Diamond53.8%77%
Confabulations20.1%17.9%
LMArena Expert12541364
SimpleQA Verified—15.3%

Multilingual o3-mini leads

Grok-2 (Dec 2024): 43.3 (#183), o3-mini: 45.9 (#159)

BenchmarkGrok-2 (Dec 2024)o3-mini
LMArena Non-English12821319
LMArena Chinese12891379
LMArena French13171333
LMArena German12871303
LMArena Japanese12441286
LMArena Korean12371314
LMArena Russian12861304
LMArena Spanish12811321

Instruction Following o3-mini leads

Grok-2 (Dec 2024): 67.1 (#197), o3-mini: 75.3 (#67)

BenchmarkGrok-2 (Dec 2024)o3-mini
LiveBench Instruction Following69.6%84.4%
LMArena Instruction Following12701338

Long Context Grok-2 (Dec 2024) leads

Grok-2 (Dec 2024): 38.7 (#186), o3-mini: 33.8 (#251)

BenchmarkGrok-2 (Dec 2024)o3-mini
LMArena Longer Query12761343
Fiction.LiveBench—50%

Writing & Preference o3-mini leads

Grok-2 (Dec 2024): 48.8 (#192), o3-mini: 50.4 (#177)

BenchmarkGrok-2 (Dec 2024)o3-mini
LMArena Text13051337
LMArena Creative Writing12841286
Short-Story Creative Writing63.6%61.7%
LMArena Multi-Turn12901320
LiveBench Language45.6%50.7%

Frequently asked questions

Is Grok-2 (Dec 2024) better than o3-mini?

o3-mini is the stronger model overall, scoring 36.7 to 33.4 on the Noometry Index.

Is Grok-2 (Dec 2024) or o3-mini better for coding?

o3-mini scores higher on coding benchmarks: 41.4 versus 32.9 in the Noometry coding category.

How many benchmarks do Grok-2 (Dec 2024) and o3-mini share?

34 benchmarks have published results for both models. Grok-2 (Dec 2024) has 34 scored results on Noometry and o3-mini has 51.

Related comparisons

Go deeper