Noometry

Model comparison

o1 vs o1-mini

o1 is the stronger model overall, scoring 40.9 to 34.0 on the Noometry Index.

Last verified . 38 shared benchmarks.

o1 OpenAI

40.9

Rank #143 Confirmed

o1-mini OpenAI

34.0

Rank #235 Confirmed

Summary

  • They share 38 benchmarks with published results for both. o1 scores higher in 8 categories and o1-mini in 1 category; 7 gaps are clear of the uncertainty.
  • The widest gap is in reasoning, where o1 leads 27.9 to 8.8.
  • The biggest single-benchmark swing is Aider Polyglot: 61.7% for o1 and 32.9% for o1-mini.

Side by side

o1o1-mini
ProviderOpenAIOpenAI
Noometry Index40.934.0
Released2024-09-122024-09-12
WeightsProprietaryProprietary
Context window200K—
Max output100K—
Input $ / M tokens$15—
Output $ / M tokens$60—
Results tracked5239

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

o1: 46.1 (#70), o1-mini: 35.5 (#224)

Benchmarko1o1-mini
Aider Polyglot61.7%32.9%
WeirdML47.6%36.3%
LiveBench Coding69.7%48%
LMArena Coding13671362
HumanEval+89%89%
MBPP+80.2%78.8%
CadEval56%—

Agentic & Tool Use Too close to call

o1: 24.6 (#117), o1-mini: 24.6 (#118)

Benchmarko1o1-mini
Cybench10%10%
METR Time Horizons51.1%—

Reasoning o1 leads

o1: 27.9 (#111), o1-mini: 8.8 (#346)

Benchmarko1o1-mini
SimpleBench41.7%18.1%
ARC-AGI-130.7%14%
LiveBench Reasoning91.6%72.3%
LMArena Hard Prompts13711333
LiveBench Data Analysis65.5%57.9%
Epoch Capabilities Index141.91135.82
LiveBench75.7%57.8%
ARC-AGI-2—0.8%
Chess Puzzles15%—
EnigmaEval5.7%—
DTBench74.7%—
LMCA22.3%—

Math Too close to call

o1: 36.1 (#175), o1-mini: 35.4 (#186)

Benchmarko1o1-mini
OTIS Mock AIME 2024-202573.3%46.9%
LiveBench Math80.3%62%
LMArena Math13881358
MATH Level 594.7%89.2%
FrontierMath (Feb 2025 set)9.3%1.7%
FrontierMath (Tiers 1-3)14.7%—

Knowledge o1 leads

o1: 41.5 (#110), o1-mini: 34.9 (#192)

Benchmarko1o1-mini
GPQA Diamond76.8%62.4%
Confabulations11.7%18.6%
LMArena Expert13611316
Humanity's Last Exam8%—
SimpleQA Verified41.1%—

Multimodal Not comparable

o1: 34.2 (#93), o1-mini: —

Benchmarko1o1-mini
LMArena Vision1168—
GeoBench80%—
VPCT37%—
SpatialViz-Bench41.4%—

Multilingual o1 leads

o1: 48.6 (#142), o1-mini: 43.6 (#182)

Benchmarko1o1-mini
LMArena Non-English13581289
LMArena Chinese13941314
LMArena French13441293
LMArena German13371278
LMArena Japanese13461245
LMArena Korean13961223
LMArena Russian13561283
LMArena Spanish13451303

Instruction Following o1 leads

o1: 74.8 (#86), o1-mini: 66.7 (#206)

Benchmarko1o1-mini
LiveBench Instruction Following81.5%65.4%
LMArena Instruction Following13671304

Long Context o1 leads

o1: 50.3 (#9), o1-mini: 40.1 (#161)

Benchmarko1o1-mini
LMArena Longer Query13781320
Fiction.LiveBench83.3%—

Writing & Preference o1 leads

o1: 55.6 (#144), o1-mini: 48.4 (#202)

Benchmarko1o1-mini
LMArena Text13661317
LMArena Creative Writing13481244
Short-Story Creative Writing70.2%64.9%
LMArena Multi-Turn13691314
LiveBench Language65.4%40.9%

Frequently asked questions

Is o1 better than o1-mini?

o1 is the stronger model overall, scoring 40.9 to 34.0 on the Noometry Index.

Is o1 or o1-mini better for coding?

o1 scores higher on coding benchmarks: 46.1 versus 35.5 in the Noometry coding category.

How many benchmarks do o1 and o1-mini share?

38 benchmarks have published results for both models. o1 has 52 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper