Noometry

Model comparison

Gemma 2 27B vs Llama 3-70B

Gemma 2 27B and Llama 3-70B score almost the same on the Noometry Index (29.2 vs 28.5), so choose on price, context window or the category you care about most.

Last verified . 25 shared benchmarks.

Gemma 2 27B Google

29.2

Rank #302 Confirmed

Llama 3-70B Meta

28.5

Rank #317 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Gemma 2 27B scores higher in 3 categories and Llama 3-70B in 5 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Gemma 2 27B leads 38.8 to 33.8.
  • The biggest single-benchmark swing is DTBench: 48% for Gemma 2 27B and 54.2% for Llama 3-70B.

Side by side

Gemma 2 27BLlama 3-70B
ProviderGoogleMeta
Noometry Index29.228.5
Released2024-06-242024-04-18
WeightsOpenOpen
Context window8K—
Max output2K—
Input $ / M tokens$0.65—
Output $ / M tokens$0.65—
Results tracked3431

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3-70B leads

Gemma 2 27B: 33.5 (#243), Llama 3-70B: 35.3 (#215)

BenchmarkGemma 2 27BLlama 3-70B
BigCodeBench Instruct42.8%43.6%
LMArena Coding12111206
BigCodeBench Complete52.5%54.5%
LiveBench Coding36%—
HumanEval+—72%
MBPP+—69%

Agentic & Tool Use Not comparable

Gemma 2 27B: —, Llama 3-70B: 20.9 (#132)

BenchmarkGemma 2 27BLlama 3-70B
Cybench—5%

Reasoning Llama 3-70B leads

Gemma 2 27B: 14.8 (#313), Llama 3-70B: 17.4 (#284)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Hard Prompts11991195
DTBench48%54.2%
Epoch Capabilities Index122.06122.91
Kagi LLM Benchmark—35.1%
LiveBench Reasoning28.1%—
LiveBench Data Analysis47.9%—
LMCA7.1%—
ForecastBench—57.1
LiveBench38.2%—
WinoGrande—83.5%

Math Llama 3-70B leads

Gemma 2 27B: 10.0 (#306), Llama 3-70B: 12.0 (#299)

BenchmarkGemma 2 27BLlama 3-70B
OTIS Mock AIME 2024-20251.4%4.3%
LMArena Math12121218
MATH Level 527.9%22.6%
LiveBench Math26.5%—

Knowledge Llama 3-70B leads

Gemma 2 27B: 18.9 (#275), Llama 3-70B: 20.8 (#272)

BenchmarkGemma 2 27BLlama 3-70B
GPQA Diamond36.5%40.6%
LMArena Expert11721149
MMLU75.7%79.3%
Confabulations27.1%—

Multilingual Gemma 2 27B leads

Gemma 2 27B: 38.8 (#221), Llama 3-70B: 33.8 (#246)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Non-English12171142
LMArena Chinese12211114
LMArena French12461231
LMArena German12091169
LMArena Japanese11751017
LMArena Korean11741017
LMArena Russian12341159
LMArena Spanish12281241

Instruction Following Llama 3-70B leads

Gemma 2 27B: 60.7 (#244), Llama 3-70B: 62.8 (#233)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Instruction Following12061194
LiveBench Instruction Following58.1%—

Long Context Gemma 2 27B leads

Gemma 2 27B: 37.3 (#213), Llama 3-70B: 35.6 (#235)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Longer Query12311174

Writing & Preference Gemma 2 27B leads

Gemma 2 27B: 44.4 (#220), Llama 3-70B: 43.0 (#225)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Text12311221
LMArena Creative Writing12411210
LMArena Multi-Turn12241223
LiveBench Language32.6%—

Frequently asked questions

Is Gemma 2 27B better than Llama 3-70B?

Gemma 2 27B and Llama 3-70B score almost the same on the Noometry Index (29.2 vs 28.5), so choose on price, context window or the category you care about most.

Is Gemma 2 27B or Llama 3-70B better for coding?

Llama 3-70B scores higher on coding benchmarks: 35.3 versus 33.5 in the Noometry coding category.

How many benchmarks do Gemma 2 27B and Llama 3-70B share?

25 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and Llama 3-70B has 31.

Related comparisons

Go deeper