Noometry

Model comparison

Claude 3 Opus vs GPT-4

Claude 3 Opus and GPT-4 score almost the same on the Noometry Index (29.5 vs 29.1), so choose on price, context window or the category you care about most.

Last verified . 32 shared benchmarks.

Claude 3 Opus Anthropic

29.5

Rank #310 Confirmed

GPT-4 OpenAI

29.1

Rank #316 Confirmed

Summary

  • They share 32 benchmarks with published results for both. Claude 3 Opus scores higher in 6 categories and GPT-4 in 2 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Claude 3 Opus leads 47.2 to 34.9.
  • The biggest single-benchmark swing is MATH Level 5: 37.5% for Claude 3 Opus and 23% for GPT-4.

Side by side

Claude 3 OpusGPT-4
ProviderAnthropicOpenAI
Noometry Index29.529.1
Released2024-02-292023-03-14
WeightsProprietaryProprietary
Context window—8K
Max output—8K
Input $ / M tokens—$30
Output $ / M tokens—$60
Results tracked4638

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3 Opus leads

Claude 3 Opus: 32.9 (#267), GPT-4: 31.6 (#283)

BenchmarkClaude 3 OpusGPT-4
WeirdML19.2%12.4%
BigCodeBench Instruct45.5%46%
LMArena Coding12641254
BigCodeBench Complete57.4%57.2%
HumanEval+77.4%79.3%
LiveBench Coding38.6%—
MBPP+73.3%—

Agentic & Tool Use Not comparable

Claude 3 Opus: 24.6 (#116), GPT-4: —

BenchmarkClaude 3 OpusGPT-4
METR Time Horizons29.5%36.1%
Cybench10%—

Reasoning GPT-4 leads

Claude 3 Opus: 14.6 (#324), GPT-4: 17.8 (#289)

BenchmarkClaude 3 OpusGPT-4
Chess Puzzles5%4%
LMArena Hard Prompts12451241
DTBench61.6%62.7%
LMCA17%17.1%
Epoch Capabilities Index126.91125.89
ForecastBench58.457.8
WinoGrande88.5%87.5%
SimpleBench23.5%—
EnigmaEval0.8%—
LiveBench Reasoning40.6%—
Mystery Game Puzzles—12%
LiveBench Data Analysis57.9%—
BIG-Bench Hard—75.1%
HellaSwag—95.3%
LiveBench49.2%—

Math Claude 3 Opus leads

Claude 3 Opus: 14.8 (#299), GPT-4: 10.8 (#309)

BenchmarkClaude 3 OpusGPT-4
OTIS Mock AIME 2024-20254.7%1.1%
LMArena Math12731269
MATH Level 537.5%23%
LiveBench Math43.6%—
GSM8K—92%

Knowledge Claude 3 Opus leads

Claude 3 Opus: 24.5 (#267), GPT-4: 18.4 (#282)

BenchmarkClaude 3 OpusGPT-4
GPQA Diamond47.2%35.7%
LMArena Expert12231211
MMLU84.6%86.4%
SimpleQA Verified12.6%—
Confabulations22.7%—
TriviaQA—84.8%

Multimodal Not comparable

Claude 3 Opus: 27.1 (#116), GPT-4: —

BenchmarkClaude 3 OpusGPT-4
LMArena Vision1023—

Multilingual Too close to call

Claude 3 Opus: 41.4 (#207), GPT-4: 40.6 (#215)

BenchmarkClaude 3 OpusGPT-4
LMArena Non-English12581246
LMArena Chinese12481242
LMArena French12751283
LMArena German12581251
LMArena Japanese12041209
LMArena Korean11871184
LMArena Russian12801251
LMArena Spanish12461261

Instruction Following GPT-4 leads

Claude 3 Opus: 64.1 (#228), GPT-4: 65.3 (#222)

BenchmarkClaude 3 OpusGPT-4
LMArena Instruction Following12481241
LiveBench Instruction Following63.9%—

Long Context Too close to call

Claude 3 Opus: 38.2 (#202), GPT-4: 37.7 (#212)

BenchmarkClaude 3 OpusGPT-4
LMArena Longer Query12591244

Writing & Preference Claude 3 Opus leads

Claude 3 Opus: 47.2 (#213), GPT-4: 34.9 (#268)

BenchmarkClaude 3 OpusGPT-4
LMArena Text12621263
LMArena Creative Writing12351244
LMArena Multi-Turn12751257
EQ-Bench Creative Writing—752
LiveBench Language50.4%—

Frequently asked questions

Is Claude 3 Opus better than GPT-4?

Claude 3 Opus and GPT-4 score almost the same on the Noometry Index (29.5 vs 29.1), so choose on price, context window or the category you care about most.

Is Claude 3 Opus or GPT-4 better for coding?

Claude 3 Opus scores higher on coding benchmarks: 32.9 versus 31.6 in the Noometry coding category.

How many benchmarks do Claude 3 Opus and GPT-4 share?

32 benchmarks have published results for both models. Claude 3 Opus has 46 scored results on Noometry and GPT-4 has 38.

Related comparisons

Go deeper