Noometry

Model comparison

Gemma 2 27B vs Llama 2-13B

Gemma 2 27B and Llama 2-13B score almost the same on the Noometry Index (29.4 vs 29.6), so choose on price, context window or the category you care about most.

Last verified . 20 shared benchmarks.

Gemma 2 27B Google

29.4

Rank #312 Confirmed

Llama 2-13B Meta

29.6

Rank #309 Confirmed

Summary

  • They share 20 benchmarks with published results for both. Gemma 2 27B scores higher in 6 categories and Llama 2-13B in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Llama 2-13B leads 31.1 to 10.7.
  • The biggest single-benchmark swing is DTBench: 48% for Gemma 2 27B and 42.2% for Llama 2-13B.

Side by side

Gemma 2 27BLlama 2-13B
ProviderGoogleMeta
Noometry Index29.429.6
Released2024-06-242023-07-18
WeightsOpenOpen
Context window8K—
Max output2K—
Input $ / M tokens$0.65—
Output $ / M tokens$0.65—
Results tracked3432

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 2 27B leads

Gemma 2 27B: 34.1 (#246), Llama 2-13B: 30.9 (#291)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Coding12111062
BigCodeBench Instruct42.8%—
LiveBench Coding36%—
BigCodeBench Complete52.5%—

Reasoning Gemma 2 27B leads

Gemma 2 27B: 15.3 (#315), Llama 2-13B: 12.8 (#337)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Hard Prompts11981051
DTBench48%42.2%
Epoch Capabilities Index122.08106.17
Chess Puzzles—0%
LiveBench Reasoning28.1%—
LiveBench Data Analysis47.9%—
LMCA7.1%—
BIG-Bench Hard—58.2%
HellaSwag—80.7%
LAMBADA—76.5%
LiveBench38.2%—
PIQA—80.8%
WinoGrande—72.8%

Math Llama 2-13B leads

Gemma 2 27B: 10.7 (#311), Llama 2-13B: 31.1 (#229)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Math12121065
OTIS Mock AIME 2024-20251.4%—
LiveBench Math26.5%—
MATH Level 527.9%—
GSM8K—36.9%

Knowledge Llama 2-13B leads

Gemma 2 27B: 19.0 (#280), Llama 2-13B: 28.1 (#249)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Expert11721030
MMLU75.7%55.6%
GPQA Diamond36.5%—
Confabulations27.1%—
ARC (AI2) Challenge—60.3%
BoolQ—82.4%
OpenBookQA—57%
TriviaQA—79.6%

Multimodal Not comparable

Gemma 2 27B: —, Llama 2-13B: —

BenchmarkGemma 2 27BLlama 2-13B
ScienceQA—55.8%

Multilingual Gemma 2 27B leads

Gemma 2 27B: 38.6 (#226), Llama 2-13B: 26.5 (#279)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Non-English12171024
LMArena Chinese12211001
LMArena French12471044
LMArena German12091009
LMArena Japanese1175894
LMArena Korean1174953
LMArena Russian12341055
LMArena Spanish12281087

Instruction Following Gemma 2 27B leads

Gemma 2 27B: 60.5 (#249), Llama 2-13B: 53.3 (#287)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Instruction Following12061045
LiveBench Instruction Following58.1%—

Long Context Gemma 2 27B leads

Gemma 2 27B: 37.3 (#218), Llama 2-13B: 32.3 (#269)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Longer Query12311064

Writing & Preference Gemma 2 27B leads

Gemma 2 27B: 44.2 (#225), Llama 2-13B: 29.8 (#289)

BenchmarkGemma 2 27BLlama 2-13B
LMArena Text12311084
LMArena Creative Writing12411047
LMArena Multi-Turn12241050
LiveBench Language32.6%—

Frequently asked questions

Is Gemma 2 27B better than Llama 2-13B?

Gemma 2 27B and Llama 2-13B score almost the same on the Noometry Index (29.4 vs 29.6), so choose on price, context window or the category you care about most.

Is Gemma 2 27B or Llama 2-13B better for coding?

Gemma 2 27B scores higher on coding benchmarks: 34.1 versus 30.9 in the Noometry coding category.

How many benchmarks do Gemma 2 27B and Llama 2-13B share?

20 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and Llama 2-13B has 32.

Related comparisons

Go deeper