Noometry

Model comparison

Gemini 2.5 Pro vs o4-mini

Gemini 2.5 Pro is the stronger model overall, scoring 45.0 to 41.6 on the Noometry Index. o4-mini costs 1.8× less per token, which makes it the better buy when Gemini 2.5 Pro's lead doesn't matter for your workload.

Last verified . 57 shared benchmarks.

Gemini 2.5 Pro Google

45.0

Rank #75 Confirmed

o4-mini OpenAI

41.6

Rank #132 Confirmed

Summary

  • They share 57 benchmarks with published results for both. Gemini 2.5 Pro scores higher in 7 categories and o4-mini in 3 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in long context, where Gemini 2.5 Pro leads 59.8 to 45.5.
  • The biggest single-benchmark swing is Omni-MATH: 41.6% for Gemini 2.5 Pro and 72% for o4-mini.
  • o4-mini is cheaper at $1.10 / $4.40 per million input/output tokens, against $1.25 / $10 for Gemini 2.5 Pro.
  • Gemini 2.5 Pro accepts more context: 1.05M tokens versus 200K.

Side by side

Gemini 2.5 Proo4-mini
ProviderGoogleOpenAI
Noometry Index45.041.6
Released2025-03-252025-04-16
WeightsProprietaryProprietary
Context window1.05M200K
Max output66K100K
Input $ / M tokens$1.25$1.10
Output $ / M tokens$10$4.40
Results tracked7860

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 2.5 Pro leads

Gemini 2.5 Pro: 42.4 (#101), o4-mini: 40.9 (#127)

BenchmarkGemini 2.5 Proo4-mini
SWE-bench Verified (bash only)53.6%45%
Aider Polyglot83.1%72%
GSO3.9%3.6%
WeirdML54%52.6%
LMArena Coding14521368
CadEval64%62%
ALE-Bench785.52826.17
AlgoTune1.511.72
SWE-bench Verified57.6%—
LMArena WebDev1227—
SciCode42.8%—
LiveBench Coding85.9%—

Agentic & Tool Use o4-mini leads

Gemini 2.5 Pro: 29.2 (#88), o4-mini: 32.6 (#61)

BenchmarkGemini 2.5 Proo4-mini
GDPval23.3%25.3%
METR Time Horizons55.4%63.9%
Terminal-Bench32.6%—
Berkeley Function Calling Leaderboard—53.2%
Remote Labor Index0.8%—
TheAgentCompany30.3%—
τ²-bench Banking13.7%—
DeepResearch Bench42.8%—
BALROG43.3%—
LMArena Search1142—
Vending-Bench 2573.64—

Reasoning Gemini 2.5 Pro leads

Gemini 2.5 Pro: 28.8 (#99), o4-mini: 24.6 (#162)

BenchmarkGemini 2.5 Proo4-mini
ARC-AGI-24.9%6.1%
SimpleBench62.4%38.7%
Kagi LLM Benchmark70.3%67.6%
ARC-AGI-141%58.7%
CritPt2%0.6%
Chess Puzzles20%26%
EnigmaEval5.6%9.2%
LMArena Hard Prompts14551351
DTBench82.4%77.6%
LMCA34.8%26.5%
Epoch Capabilities Index145.32145.64
ForecastBench61.361.8
LiveBench Reasoning89.8%—
Mystery Game Puzzles—5%
LiveBench Data Analysis79.9%—
LiveBench82.3%—

Math o4-mini leads

Gemini 2.5 Pro: 32.5 (#213), o4-mini: 40.8 (#89)

BenchmarkGemini 2.5 Proo4-mini
FrontierMath (Tiers 1-3)24.6%36.1%
FrontierMath Tier 40%4.9%
OTIS Mock AIME 2024-202584.7%81.7%
Omni-MATH41.6%72%
LMArena Math14501389
MATH Level 595.9%97.8%
FrontierMath (Feb 2025 set)14.1%24.8%
FrontierMath Tier 4 (v1)4.2%6.3%
LiveBench Math90.2%—

Knowledge Gemini 2.5 Pro leads

Gemini 2.5 Pro: 56.0 (#46), o4-mini: 43.6 (#91)

BenchmarkGemini 2.5 Proo4-mini
GPQA Diamond85.3%79.6%
Humanity's Last Exam21.6%18.1%
MMLU-Pro86.3%82%
Confabulations10.6%15.8%
Vectara Hallucination Rate7%18.6%
GPQA (HELM)74.9%73.5%
LMArena Expert14521343
SimpleQA Verified—19.6%

Multimodal Gemini 2.5 Pro leads

Gemini 2.5 Pro: 45.2 (#18), o4-mini: 40.2 (#49)

BenchmarkGemini 2.5 Proo4-mini
LMArena Vision12631194
GeoBench86%64%
VPCT48%57.5%
LMArena Document1421—
SpatialViz-Bench44.7%—

Multilingual Gemini 2.5 Pro leads

Gemini 2.5 Pro: 55.3 (#31), o4-mini: 47.0 (#154)

BenchmarkGemini 2.5 Proo4-mini
LMArena Non-English14511337
LMArena Chinese15071354
LMArena French14721364
LMArena German14871336
LMArena Japanese14611308
LMArena Korean14341312
LMArena Russian14611334
LMArena Spanish14731347

Instruction Following Too close to call

Gemini 2.5 Pro: 75.0 (#75), o4-mini: 75.2 (#68)

BenchmarkGemini 2.5 Proo4-mini
IFEval84%92.8%
LMArena Instruction Following14371321
LiveBench Instruction Following80.6%—

Long Context Gemini 2.5 Pro leads

Gemini 2.5 Pro: 59.8 (#5), o4-mini: 45.5 (#33)

BenchmarkGemini 2.5 Proo4-mini
Fiction.LiveBench91.7%77.8%
LMArena Longer Query14491315

Writing & Preference Gemini 2.5 Pro leads

Gemini 2.5 Pro: 63.7 (#62), o4-mini: 54.0 (#152)

BenchmarkGemini 2.5 Proo4-mini
LMArena Text14581353
LMArena Creative Writing14541294
Short-Story Creative Writing83.8%75%
WildBench85.7%85.4%
LMArena Multi-Turn14531350
EQ-Bench Creative Writing1421—
LiveBench Language67.8%—

Frequently asked questions

Is Gemini 2.5 Pro better than o4-mini?

Gemini 2.5 Pro is the stronger model overall, scoring 45.0 to 41.6 on the Noometry Index. o4-mini costs 1.8× less per token, which makes it the better buy when Gemini 2.5 Pro's lead doesn't matter for your workload.

Which is cheaper, Gemini 2.5 Pro or o4-mini?

o4-mini is cheaper. It lists at $1.10 per million input tokens and $4.40 per million output tokens; Gemini 2.5 Pro lists at $1.25 and $10.

Is Gemini 2.5 Pro or o4-mini better for coding?

Gemini 2.5 Pro scores higher on coding benchmarks: 42.4 versus 40.9 in the Noometry coding category.

Which has the bigger context window?

Gemini 2.5 Pro does, with 1.05M tokens against 200K.

How many benchmarks do Gemini 2.5 Pro and o4-mini share?

57 benchmarks have published results for both models. Gemini 2.5 Pro has 78 scored results on Noometry and o4-mini has 60.

Related comparisons

Go deeper