Noometry

Model comparison

Gemma 2 9B vs GPT-4 Turbo

GPT-4 Turbo is the stronger model overall, scoring 30.5 to 25.9 on the Noometry Index.

Last verified . 24 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

GPT-4 Turbo OpenAI

30.5

Rank #292 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Gemma 2 9B scores higher in 2 categories and GPT-4 Turbo in 6 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where GPT-4 Turbo leads 47.7 to 32.1.
  • The biggest single-benchmark swing is MATH Level 5: 21% for Gemma 2 9B and 46.7% for GPT-4 Turbo.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9BGPT-4 Turbo
ProviderGoogleOpenAI
Noometry Index25.930.5
Released2024-06-242023-11-06
WeightsOpenProprietary
Context window—128K
Max output—4K
Input $ / M tokens—$10
Output $ / M tokens—$30
Results tracked3536

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-4 Turbo leads

Gemma 2 9B: 29.4 (#304), GPT-4 Turbo: 33.8 (#249)

BenchmarkGemma 2 9BGPT-4 Turbo
BigCodeBench Instruct34.7%48.2%
LMArena Coding11731268
BigCodeBench Complete40.6%58.2%
WeirdML—18%
LiveBench Coding22.5%—
HumanEval+—86.6%
MBPP+—73.3%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, GPT-4 Turbo: —

BenchmarkGemma 2 9BGPT-4 Turbo
METR Time Horizons—36.7%

Reasoning Too close to call

Gemma 2 9B: 15.9 (#309), GPT-4 Turbo: 15.3 (#317)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Hard Prompts11711251
Epoch Capabilities Index119.83127.25
SimpleBench—25.1%
Chess Puzzles—6%
LiveBench Reasoning15.2%—
DTBench—61.6%
LiveBench Data Analysis36.4%—
LMCA—9.8%
ForecastBench—59.4
LiveBench28.7%—
PIQA83.7%—

Math Too close to call

Gemma 2 9B: 9.9 (#318), GPT-4 Turbo: 9.0 (#322)

BenchmarkGemma 2 9BGPT-4 Turbo
OTIS Mock AIME 2024-20250.6%6.7%
LMArena Math11831272
MATH Level 521%46.7%
FrontierMath (Tiers 1-3)—0.7%
LiveBench Math19.8%—
GSM8K84.9%—

Knowledge GPT-4 Turbo leads

Gemma 2 9B: 9.7 (#305), GPT-4 Turbo: 24.3 (#268)

BenchmarkGemma 2 9BGPT-4 Turbo
GPQA Diamond27.5%46.6%
LMArena Expert11471223
MMLU72.1%81.3%
Confabulations—28.4%
BoolQ85.7%—

Multimodal Not comparable

Gemma 2 9B: —, GPT-4 Turbo: 30.6 (#110)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Vision—1090

Multilingual GPT-4 Turbo leads

Gemma 2 9B: 36.6 (#238), GPT-4 Turbo: 40.5 (#216)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Non-English11881245
LMArena Chinese11851242
LMArena French11901276
LMArena German11861259
LMArena Japanese11441194
LMArena Korean11371187
LMArena Russian12001259
LMArena Spanish12001260

Instruction Following GPT-4 Turbo leads

Gemma 2 9B: 57.6 (#269), GPT-4 Turbo: 65.8 (#216)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Instruction Following11781249
LiveBench Instruction Following52.6%—

Long Context GPT-4 Turbo leads

Gemma 2 9B: 36.3 (#233), GPT-4 Turbo: 38.0 (#206)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Longer Query11971254

Writing & Preference GPT-4 Turbo leads

Gemma 2 9B: 32.1 (#281), GPT-4 Turbo: 47.7 (#206)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Text12071272
LMArena Creative Writing12061269
LMArena Multi-Turn11931267
EQ-Bench Creative Writing841—
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than GPT-4 Turbo?

GPT-4 Turbo is the stronger model overall, scoring 30.5 to 25.9 on the Noometry Index.

Is Gemma 2 9B or GPT-4 Turbo better for coding?

GPT-4 Turbo scores higher on coding benchmarks: 33.8 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and GPT-4 Turbo share?

24 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and GPT-4 Turbo has 36.

Related comparisons

Go deeper