Noometry

Model comparison

Llama 3.1-405B vs o1-mini

o1-mini is the stronger model overall, scoring 33.7 to 30.4 on the Noometry Index.

Last verified . 25 shared benchmarks.

Llama 3.1-405B Meta

30.4

Rank #279 Confirmed

o1-mini OpenAI

33.7

Rank #230 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Llama 3.1-405B scores higher in 1 category and o1-mini in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1-mini leads 33.7 to 17.3.
  • The biggest single-benchmark swing is MATH Level 5: 49.8% for Llama 3.1-405B and 89.2% for o1-mini.
  • Llama 3.1-405B has downloadable open weights; the other is API-only.

Side by side

Llama 3.1-405Bo1-mini
ProviderMetaOpenAI
Noometry Index30.433.7
Released2024-07-232024-09-12
WeightsOpenProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4239

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

Llama 3.1-405B: 32.7 (#257), o1-mini: 35.1 (#218)

BenchmarkLlama 3.1-405Bo1-mini
WeirdML21.4%36.3%
LMArena Coding12911362
Aider Polyglot—32.9%
LiveBench Coding—48%
HumanEval+—89%
MBPP+—78.8%

Agentic & Tool Use o1-mini leads

Llama 3.1-405B: 20.9 (#131), o1-mini: 24.3 (#112)

BenchmarkLlama 3.1-405Bo1-mini
Cybench7.5%10%
TheAgentCompany7.4%—

Reasoning Llama 3.1-405B leads

Llama 3.1-405B: 16.4 (#293), o1-mini: 8.9 (#340)

BenchmarkLlama 3.1-405Bo1-mini
SimpleBench23%18.1%
LMArena Hard Prompts12701333
Epoch Capabilities Index128.76135.82
ARC-AGI-2—0.8%
Kagi LLM Benchmark45%—
ARC-AGI-1—14%
LiveBench Reasoning—72.3%
DTBench61.4%—
LiveBench Data Analysis—57.9%
BIG-Bench Hard82.9%—
ForecastBench59.9—
HellaSwag89.2%—
LiveBench—57.8%
PIQA85.9%—
WinoGrande89.2%—

Math o1-mini leads

Llama 3.1-405B: 17.3 (#283), o1-mini: 33.7 (#168)

BenchmarkLlama 3.1-405Bo1-mini
OTIS Mock AIME 2024-20259.7%46.9%
LMArena Math12811358
MATH Level 549.8%89.2%
Omni-MATH24.9%—
LiveBench Math—62%
FrontierMath (Feb 2025 set)—1.7%

Knowledge o1-mini leads

Llama 3.1-405B: 30.4 (#221), o1-mini: 34.8 (#180)

BenchmarkLlama 3.1-405Bo1-mini
GPQA Diamond50.9%62.4%
Confabulations17.6%18.6%
LMArena Expert12431316
MMLU-Pro72.3%—
GPQA (HELM)52.2%—
ARC (AI2) Challenge95.3%—
MMLU84.5%—
TriviaQA82.7%—

Multilingual o1-mini leads

Llama 3.1-405B: 40.9 (#209), o1-mini: 43.8 (#177)

BenchmarkLlama 3.1-405Bo1-mini
LMArena Non-English12481289
LMArena Chinese12421314
LMArena French12781292
LMArena German12521278
LMArena Japanese12081245
LMArena Korean11841223
LMArena Russian12651283
LMArena Spanish12591302

Instruction Following Too close to call

Llama 3.1-405B: 66.1 (#209), o1-mini: 67.0 (#200)

BenchmarkLlama 3.1-405Bo1-mini
LMArena Instruction Following12591304
LiveBench Instruction Following—65.4%
IFEval81.1%—

Long Context o1-mini leads

Llama 3.1-405B: 38.4 (#192), o1-mini: 40.1 (#156)

BenchmarkLlama 3.1-405Bo1-mini
LMArena Longer Query12661321

Writing & Preference o1-mini leads

Llama 3.1-405B: 39.1 (#246), o1-mini: 48.6 (#196)

BenchmarkLlama 3.1-405Bo1-mini
LMArena Text12841317
LMArena Creative Writing12621244
LMArena Multi-Turn12971314
Short-Story Creative Writing—64.9%
EQ-Bench Creative Writing870—
WildBench78.3%—
LiveBench Language—40.9%

Frequently asked questions

Is Llama 3.1-405B better than o1-mini?

o1-mini is the stronger model overall, scoring 33.7 to 30.4 on the Noometry Index.

Is Llama 3.1-405B or o1-mini better for coding?

o1-mini scores higher on coding benchmarks: 35.1 versus 32.7 in the Noometry coding category.

How many benchmarks do Llama 3.1-405B and o1-mini share?

25 benchmarks have published results for both models. Llama 3.1-405B has 42 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper