Noometry

Model comparison

GPT-4 vs Llama 3-8B

GPT-4 is the stronger model overall, scoring 28.9 to 25.3 on the Noometry Index. Llama 3-8B costs 268× less per token, which makes it the better buy when GPT-4's lead doesn't matter for your workload.

Last verified . 30 shared benchmarks.

GPT-4 OpenAI

28.9

Rank #308 Confirmed

Llama 3-8B Meta

25.3

Rank #338 Confirmed

Summary

  • They share 30 benchmarks with published results for both. GPT-4 scores higher in 7 categories and Llama 3-8B in 1 category; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where GPT-4 leads 18.4 to 7.8.
  • The biggest single-benchmark swing is BigCodeBench Complete: 57.2% for GPT-4 and 36.9% for Llama 3-8B.
  • Llama 3-8B is cheaper at $0.14 / $0.14 per million input/output tokens, against $30 / $60 for GPT-4.
  • Llama 3-8B has downloadable open weights; the other is API-only.

Side by side

GPT-4Llama 3-8B
ProviderOpenAIMeta
Noometry Index28.925.3
Released2023-03-142024-04-18
WeightsProprietaryOpen
Context window8K8K
Max output8K8K
Input $ / M tokens$30$0.14
Output $ / M tokens$60$0.14
Results tracked3834

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

GPT-4: 31.2 (#277), Llama 3-8B: 30.5 (#285)

BenchmarkGPT-4Llama 3-8B
BigCodeBench Instruct46%31.9%
LMArena Coding12541152
BigCodeBench Complete57.2%36.9%
HumanEval+79.3%56.7%
WeirdML12.4%—
MBPP+—54.8%

Agentic & Tool Use Not comparable

GPT-4: —, Llama 3-8B: —

BenchmarkGPT-4Llama 3-8B
METR Time Horizons36.1%—

Reasoning GPT-4 leads

GPT-4: 17.3 (#285), Llama 3-8B: 13.8 (#321)

BenchmarkGPT-4Llama 3-8B
Chess Puzzles4%0%
LMArena Hard Prompts12411133
DTBench62.7%43.9%
Epoch Capabilities Index125.88116.33
ForecastBench57.858.6
WinoGrande87.5%75.7%
Mystery Game Puzzles12%—
LMCA17.1%—
Adversarial NLI—57.3%
BIG-Bench Hard75.1%—
HellaSwag95.3%—

Math GPT-4 leads

GPT-4: 10.1 (#304), Llama 3-8B: 8.3 (#317)

BenchmarkGPT-4Llama 3-8B
OTIS Mock AIME 2024-20251.1%1.9%
LMArena Math12691151
MATH Level 523%6.1%
GSM8K92%—

Knowledge GPT-4 leads

GPT-4: 18.4 (#277), Llama 3-8B: 7.8 (#303)

BenchmarkGPT-4Llama 3-8B
GPQA Diamond35.7%26.1%
LMArena Expert12111113
MMLU86.4%68.8%
TriviaQA84.8%67.7%
ARC (AI2) Challenge—82.8%
OpenBookQA—82.6%

Multilingual GPT-4 leads

GPT-4: 40.8 (#210), Llama 3-8B: 31.0 (#256)

BenchmarkGPT-4Llama 3-8B
LMArena Non-English12461098
LMArena Chinese12421076
LMArena French12821158
LMArena German12511104
LMArena Japanese1209967
LMArena Korean11841003
LMArena Russian12511109
LMArena Spanish12611172

Instruction Following GPT-4 leads

GPT-4: 65.5 (#217), Llama 3-8B: 58.7 (#255)

BenchmarkGPT-4Llama 3-8B
LMArena Instruction Following12411127

Long Context GPT-4 leads

GPT-4: 37.7 (#207), Llama 3-8B: 34.2 (#246)

BenchmarkGPT-4Llama 3-8B
LMArena Longer Query12441128

Writing & Preference Llama 3-8B leads

GPT-4: 35.0 (#263), Llama 3-8B: 37.7 (#251)

BenchmarkGPT-4Llama 3-8B
LMArena Text12631166
LMArena Creative Writing12441150
LMArena Multi-Turn12571152
EQ-Bench Creative Writing752—

Frequently asked questions

Is GPT-4 better than Llama 3-8B?

GPT-4 is the stronger model overall, scoring 28.9 to 25.3 on the Noometry Index. Llama 3-8B costs 268× less per token, which makes it the better buy when GPT-4's lead doesn't matter for your workload.

Which is cheaper, GPT-4 or Llama 3-8B?

Llama 3-8B is cheaper. It lists at $0.14 per million input tokens and $0.14 per million output tokens; GPT-4 lists at $30 and $60.

Is GPT-4 or Llama 3-8B better for coding?

They score almost the same on coding (31.2 vs 30.5); test both on your own repository before choosing.

Which has the bigger context window?

Both accept 8K tokens.

How many benchmarks do GPT-4 and Llama 3-8B share?

30 benchmarks have published results for both models. GPT-4 has 38 scored results on Noometry and Llama 3-8B has 34.

Related comparisons

Go deeper