Noometry

Model comparison

Grok-2 (Dec 2024) vs o1

o1 is the stronger model overall, scoring 40.6 to 33.4 on the Noometry Index.

Last verified . 34 shared benchmarks.

Grok-2 (Dec 2024) xAI

33.4

Rank #233 Confirmed

o1 OpenAI

40.6

Rank #140 Confirmed

Summary

  • They share 34 benchmarks with published results for both. Grok-2 (Dec 2024) scores higher in 0 categories and o1 in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1 leads 34.9 to 19.6.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 11.5% for Grok-2 (Dec 2024) and 73.3% for o1.

Side by side

Grok-2 (Dec 2024)o1
ProviderxAIOpenAI
Noometry Index33.440.6
Released2024-08-132024-09-12
WeightsProprietaryProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$15
Output $ / M tokens—$60
Results tracked3452

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

Grok-2 (Dec 2024): 32.9 (#254), o1: 45.7 (#64)

BenchmarkGrok-2 (Dec 2024)o1
WeirdML22.2%47.6%
LiveBench Coding46.4%69.7%
LMArena Coding12881367
Aider Polyglot—61.7%
CadEval—56%
HumanEval+—89%
MBPP+—80.2%

Agentic & Tool Use Not comparable

Grok-2 (Dec 2024): —, o1: 24.3 (#111)

BenchmarkGrok-2 (Dec 2024)o1
Cybench—10%
METR Time Horizons—51.1%

Reasoning o1 leads

Grok-2 (Dec 2024): 16.4 (#294), o1: 27.5 (#103)

BenchmarkGrok-2 (Dec 2024)o1
SimpleBench22.7%41.7%
LiveBench Reasoning54.8%91.6%
LMArena Hard Prompts12721372
DTBench65.2%74.7%
LiveBench Data Analysis54.5%65.5%
Epoch Capabilities Index130.48141.92
LiveBench54.3%75.7%
ARC-AGI-1—30.7%
Chess Puzzles—15%
EnigmaEval—5.7%
LMCA—22.3%

Math o1 leads

Grok-2 (Dec 2024): 19.6 (#278), o1: 34.9 (#150)

BenchmarkGrok-2 (Dec 2024)o1
OTIS Mock AIME 2024-202511.5%73.3%
LiveBench Math54.9%80.3%
LMArena Math12831388
MATH Level 563.5%94.7%
FrontierMath (Feb 2025 set)0.7%9.3%
FrontierMath (Tiers 1-3)—14.7%

Knowledge o1 leads

Grok-2 (Dec 2024): 29.8 (#225), o1: 41.5 (#101)

BenchmarkGrok-2 (Dec 2024)o1
GPQA Diamond53.8%76.8%
Confabulations20.1%11.7%
LMArena Expert12541362
Humanity's Last Exam—8%
SimpleQA Verified—41.1%

Multimodal Not comparable

Grok-2 (Dec 2024): —, o1: 33.6 (#87)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Vision—1168
GeoBench—80%
VPCT—37%
SpatialViz-Bench—41.4%

Multilingual o1 leads

Grok-2 (Dec 2024): 43.3 (#183), o1: 48.8 (#137)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Non-English12821359
LMArena Chinese12891394
LMArena French13171343
LMArena German12871337
LMArena Japanese12441346
LMArena Korean12371396
LMArena Russian12861356
LMArena Spanish12811345

Instruction Following o1 leads

Grok-2 (Dec 2024): 67.1 (#197), o1: 75.0 (#82)

BenchmarkGrok-2 (Dec 2024)o1
LiveBench Instruction Following69.6%81.5%
LMArena Instruction Following12701368

Long Context o1 leads

Grok-2 (Dec 2024): 38.7 (#186), o1: 50.3 (#9)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Longer Query12761378
Fiction.LiveBench—83.3%

Writing & Preference o1 leads

Grok-2 (Dec 2024): 48.8 (#192), o1: 55.8 (#139)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Text13051366
LMArena Creative Writing12841347
Short-Story Creative Writing63.6%70.2%
LMArena Multi-Turn12901368
LiveBench Language45.6%65.4%

Frequently asked questions

Is Grok-2 (Dec 2024) better than o1?

o1 is the stronger model overall, scoring 40.6 to 33.4 on the Noometry Index.

Is Grok-2 (Dec 2024) or o1 better for coding?

o1 scores higher on coding benchmarks: 45.7 versus 32.9 in the Noometry coding category.

How many benchmarks do Grok-2 (Dec 2024) and o1 share?

34 benchmarks have published results for both models. Grok-2 (Dec 2024) has 34 scored results on Noometry and o1 has 52.

Related comparisons

Go deeper