Noometry

Model comparison

Claude 3 Opus vs Gemma 1.1 2b IT

Claude 3 Opus and Gemma 1.1 2b IT score almost the same on the Noometry Index (29.5 vs 29.3), so choose on price, context window or the category you care about most.

Last verified . 16 shared benchmarks.

Claude 3 Opus Anthropic

29.5

Rank #310 Confirmed

Gemma 1.1 2b IT Google

29.3

Rank #313 Confirmed

Summary

  • They share 16 benchmarks with published results for both. Claude 3 Opus scores higher in 5 categories and Gemma 1.1 2b IT in 3 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Claude 3 Opus leads 47.2 to 25.1.
  • Gemma 1.1 2b IT has downloadable open weights; the other is API-only.

Side by side

Claude 3 OpusGemma 1.1 2b IT
ProviderAnthropicGoogle
Noometry Index29.529.3
Released2024-02-29—
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4616

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3 Opus leads

Claude 3 Opus: 32.9 (#267), Gemma 1.1 2b IT: 30.1 (#299)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Coding12641034
HumanEval+77.4%17.7%
MBPP+73.3%23.3%
WeirdML19.2%—
BigCodeBench Instruct45.5%—
LiveBench Coding38.6%—
BigCodeBench Complete57.4%—

Agentic & Tool Use Not comparable

Claude 3 Opus: 24.6 (#116), Gemma 1.1 2b IT: —

BenchmarkClaude 3 OpusGemma 1.1 2b IT
Cybench10%—
METR Time Horizons29.5%—

Reasoning Gemma 1.1 2b IT leads

Claude 3 Opus: 14.6 (#324), Gemma 1.1 2b IT: 19.1 (#270)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Hard Prompts12451005
SimpleBench23.5%—
Chess Puzzles5%—
EnigmaEval0.8%—
LiveBench Reasoning40.6%—
DTBench61.6%—
LiveBench Data Analysis57.9%—
LMCA17%—
Epoch Capabilities Index126.91—
ForecastBench58.4—
LiveBench49.2%—
WinoGrande88.5%—

Math Gemma 1.1 2b IT leads

Claude 3 Opus: 14.8 (#299), Gemma 1.1 2b IT: 30.8 (#232)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Math12731047
OTIS Mock AIME 2024-20254.7%—
LiveBench Math43.6%—
MATH Level 537.5%—

Knowledge Gemma 1.1 2b IT leads

Claude 3 Opus: 24.5 (#267), Gemma 1.1 2b IT: 26.5 (#258)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Expert1223970
GPQA Diamond47.2%—
SimpleQA Verified12.6%—
Confabulations22.7%—
MMLU84.6%—

Multimodal Not comparable

Claude 3 Opus: 27.1 (#116), Gemma 1.1 2b IT: —

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Vision1023—

Multilingual Claude 3 Opus leads

Claude 3 Opus: 41.4 (#207), Gemma 1.1 2b IT: 24.6 (#289)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Non-English1258988
LMArena Chinese12481012
LMArena German1258944
LMArena Korean1187899
LMArena Russian1280990
LMArena French1275—
LMArena Japanese1204—
LMArena Spanish1246—

Instruction Following Claude 3 Opus leads

Claude 3 Opus: 64.1 (#228), Gemma 1.1 2b IT: 49.9 (#299)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Instruction Following1248992
LiveBench Instruction Following63.9%—

Long Context Claude 3 Opus leads

Claude 3 Opus: 38.2 (#202), Gemma 1.1 2b IT: 30.6 (#286)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Longer Query12591003

Writing & Preference Claude 3 Opus leads

Claude 3 Opus: 47.2 (#213), Gemma 1.1 2b IT: 25.1 (#306)

BenchmarkClaude 3 OpusGemma 1.1 2b IT
LMArena Text12621022
LMArena Creative Writing1235998
LMArena Multi-Turn1275959
LiveBench Language50.4%—

Frequently asked questions

Is Claude 3 Opus better than Gemma 1.1 2b IT?

Claude 3 Opus and Gemma 1.1 2b IT score almost the same on the Noometry Index (29.5 vs 29.3), so choose on price, context window or the category you care about most.

Is Claude 3 Opus or Gemma 1.1 2b IT better for coding?

Claude 3 Opus scores higher on coding benchmarks: 32.9 versus 30.1 in the Noometry coding category.

How many benchmarks do Claude 3 Opus and Gemma 1.1 2b IT share?

16 benchmarks have published results for both models. Claude 3 Opus has 46 scored results on Noometry and Gemma 1.1 2b IT has 16.

Related comparisons

Go deeper