Noometry

Model comparison

Gemma 2 9B vs Phi 3 Mini 4k Instruct

Phi 3 Mini 4k Instruct is the stronger model overall, scoring 27.4 to 25.7 on the Noometry Index.

Last verified . 25 shared benchmarks.

Gemma 2 9B Google

25.7

Rank #334 Confirmed

Phi 3 Mini 4k Instruct Microsoft

27.4

Rank #324 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Gemma 2 9B scores higher in 6 categories and Phi 3 Mini 4k Instruct in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Phi 3 Mini 4k Instruct leads 27.9 to 9.7.
  • The biggest single-benchmark swing is LiveBench Language: 25.5% for Gemma 2 9B and 9.2% for Phi 3 Mini 4k Instruct.

Side by side

Gemma 2 9BPhi 3 Mini 4k Instruct
ProviderGoogleMicrosoft
Noometry Index25.727.4
Released2024-06-242024-04-23
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3535

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 2 9B leads

Gemma 2 9B: 29.0 (#299), Phi 3 Mini 4k Instruct: 26.1 (#319)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LiveBench Coding22.5%15.5%
LMArena Coding11741093
BigCodeBench Instruct34.7%—
BigCodeBench Complete40.6%—
HumanEval+—59.1%
MBPP+—54.2%

Reasoning Gemma 2 9B leads

Gemma 2 9B: 15.3 (#306), Phi 3 Mini 4k Instruct: 13.6 (#326)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LiveBench Reasoning15.2%26.8%
LMArena Hard Prompts11721072
LiveBench Data Analysis36.4%34.7%
LiveBench28.7%22.4%
Chess Puzzles—0%
Adversarial NLI—52.8%
BIG-Bench Hard—71.7%
Epoch Capabilities Index119.78—
HellaSwag—76.7%
PIQA83.7%—
WinoGrande—70.8%

Math Phi 3 Mini 4k Instruct leads

Gemma 2 9B: 9.3 (#312), Phi 3 Mini 4k Instruct: 24.5 (#258)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LiveBench Math19.8%15.7%
LMArena Math11831111
OTIS Mock AIME 2024-20250.6%—
MATH Level 521%—
GSM8K84.9%—

Knowledge Phi 3 Mini 4k Instruct leads

Gemma 2 9B: 9.7 (#300), Phi 3 Mini 4k Instruct: 27.9 (#241)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LMArena Expert11471045
MMLU72.1%68.8%
GPQA Diamond27.5%—
ARC (AI2) Challenge—84.9%
BoolQ85.7%—
OpenBookQA—88%
TriviaQA—64%

Multilingual Gemma 2 9B leads

Gemma 2 9B: 36.8 (#233), Phi 3 Mini 4k Instruct: 26.5 (#275)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LMArena Non-English11881021
LMArena Chinese11851021
LMArena French11891075
LMArena German11861044
LMArena Japanese1144935
LMArena Korean1137905
LMArena Russian12001022
LMArena Spanish12001084

Instruction Following Gemma 2 9B leads

Gemma 2 9B: 57.8 (#264), Phi 3 Mini 4k Instruct: 48.0 (#298)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LiveBench Instruction Following52.6%39.1%
LMArena Instruction Following11781053

Long Context Gemma 2 9B leads

Gemma 2 9B: 36.3 (#228), Phi 3 Mini 4k Instruct: 31.7 (#271)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LMArena Longer Query11971044

Writing & Preference Gemma 2 9B leads

Gemma 2 9B: 32.3 (#276), Phi 3 Mini 4k Instruct: 27.7 (#295)

BenchmarkGemma 2 9BPhi 3 Mini 4k Instruct
LMArena Text12071073
LMArena Creative Writing12061037
LMArena Multi-Turn11931018
LiveBench Language25.5%9.2%
EQ-Bench Creative Writing841—

Frequently asked questions

Is Gemma 2 9B better than Phi 3 Mini 4k Instruct?

Phi 3 Mini 4k Instruct is the stronger model overall, scoring 27.4 to 25.7 on the Noometry Index.

Is Gemma 2 9B or Phi 3 Mini 4k Instruct better for coding?

Gemma 2 9B scores higher on coding benchmarks: 29.0 versus 26.1 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Phi 3 Mini 4k Instruct share?

25 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Phi 3 Mini 4k Instruct has 35.

Related comparisons

Go deeper