Noometry

Model comparison

Olmo 2 0325 32b Instruct vs Phi 3 Medium 4k Instruct

Olmo 2 0325 32b Instruct and Phi 3 Medium 4k Instruct score almost the same on the Noometry Index (32.7 vs 33.0), so choose on price, context window or the category you care about most.

Last verified . 11 shared benchmarks.

Olmo 2 0325 32b Instruct Allen Institute for AI (Ai2)

32.7

Rank #254 Confirmed

Phi 3 Medium 4k Instruct Microsoft

33.0

Rank #250 Confirmed

Summary

  • They share 11 benchmarks with published results for both. Olmo 2 0325 32b Instruct scores higher in 6 categories and Phi 3 Medium 4k Instruct in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Phi 3 Medium 4k Instruct leads 30.2 to 19.5.

Side by side

Olmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
ProviderAllen Institute for AI (Ai2)Microsoft
Noometry Index32.733.0
Released——
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1617

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Olmo 2 0325 32b Instruct leads

Olmo 2 0325 32b Instruct: 35.2 (#227), Phi 3 Medium 4k Instruct: 32.9 (#268)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Coding12101130

Reasoning Olmo 2 0325 32b Instruct leads

Olmo 2 0325 32b Instruct: 23.6 (#175), Phi 3 Medium 4k Instruct: 21.7 (#214)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Hard Prompts12081127

Math Phi 3 Medium 4k Instruct leads

Olmo 2 0325 32b Instruct: 26.8 (#255), Phi 3 Medium 4k Instruct: 33.4 (#202)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Math12081173
Omni-MATH16.1%—

Knowledge Phi 3 Medium 4k Instruct leads

Olmo 2 0325 32b Instruct: 19.5 (#279), Phi 3 Medium 4k Instruct: 30.2 (#229)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
MMLU-Pro41.4%—
GPQA (HELM)28.7%—
LMArena Expert—1108

Multilingual Olmo 2 0325 32b Instruct leads

Olmo 2 0325 32b Instruct: 34.8 (#248), Phi 3 Medium 4k Instruct: 30.6 (#263)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Non-English11601094
LMArena Chinese11921108
LMArena Russian11871145
LMArena French—1110
LMArena German—1101
LMArena Japanese—1042
LMArena Korean—954
LMArena Spanish—1095

Instruction Following Olmo 2 0325 32b Instruct leads

Olmo 2 0325 32b Instruct: 61.5 (#244), Phi 3 Medium 4k Instruct: 57.6 (#268)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Instruction Following11861113
IFEval78%—

Long Context Olmo 2 0325 32b Instruct leads

Olmo 2 0325 32b Instruct: 36.2 (#234), Phi 3 Medium 4k Instruct: 34.0 (#255)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Longer Query11941120

Writing & Preference Olmo 2 0325 32b Instruct leads

Olmo 2 0325 32b Instruct: 42.1 (#236), Phi 3 Medium 4k Instruct: 34.1 (#272)

BenchmarkOlmo 2 0325 32b InstructPhi 3 Medium 4k Instruct
LMArena Text12181138
LMArena Creative Writing11991107
LMArena Multi-Turn12211088
WildBench73.4%—

Frequently asked questions

Is Olmo 2 0325 32b Instruct better than Phi 3 Medium 4k Instruct?

Olmo 2 0325 32b Instruct and Phi 3 Medium 4k Instruct score almost the same on the Noometry Index (32.7 vs 33.0), so choose on price, context window or the category you care about most.

Is Olmo 2 0325 32b Instruct or Phi 3 Medium 4k Instruct better for coding?

Olmo 2 0325 32b Instruct scores higher on coding benchmarks: 35.2 versus 32.9 in the Noometry coding category.

How many benchmarks do Olmo 2 0325 32b Instruct and Phi 3 Medium 4k Instruct share?

11 benchmarks have published results for both models. Olmo 2 0325 32b Instruct has 16 scored results on Noometry and Phi 3 Medium 4k Instruct has 17.

Related comparisons

Go deeper