Noometry

Model comparison

Codellama 34b Instruct vs GPT-4

Codellama 34b Instruct is the stronger model overall, scoring 30.8 to 29.1 on the Noometry Index.

Last verified . 13 shared benchmarks.

Codellama 34b Instruct Meta

30.8

Rank #287 Confirmed

GPT-4 OpenAI

29.1

Rank #316 Confirmed

Summary

  • They share 13 benchmarks with published results for both. Codellama 34b Instruct scores higher in 2 categories and GPT-4 in 5 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Codellama 34b Instruct leads 31.0 to 10.8.
  • The biggest single-benchmark swing is BigCodeBench Complete: 37.1% for Codellama 34b Instruct and 57.2% for GPT-4.
  • Codellama 34b Instruct has downloadable open weights; the other is API-only.

Side by side

Codellama 34b InstructGPT-4
ProviderMetaOpenAI
Noometry Index30.829.1
Released—2023-03-14
WeightsOpenProprietary
Context window—8K
Max output—8K
Input $ / M tokens—$30
Output $ / M tokens—$60
Results tracked1438

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-4 leads

Codellama 34b Instruct: 28.5 (#314), GPT-4: 31.6 (#283)

BenchmarkCodellama 34b InstructGPT-4
BigCodeBench Instruct29%46%
LMArena Coding10461254
BigCodeBench Complete37.1%57.2%
HumanEval+43.9%79.3%
WeirdML—12.4%
MBPP+56.3%—

Agentic & Tool Use Not comparable

Codellama 34b Instruct: —, GPT-4: —

BenchmarkCodellama 34b InstructGPT-4
METR Time Horizons—36.1%

Reasoning Codellama 34b Instruct leads

Codellama 34b Instruct: 19.6 (#255), GPT-4: 17.8 (#289)

BenchmarkCodellama 34b InstructGPT-4
LMArena Hard Prompts10321241
Chess Puzzles—4%
Mystery Game Puzzles—12%
DTBench—62.7%
LMCA—17.1%
BIG-Bench Hard—75.1%
Epoch Capabilities Index—125.89
ForecastBench—57.8
HellaSwag—95.3%
WinoGrande—87.5%

Math Codellama 34b Instruct leads

Codellama 34b Instruct: 31.0 (#230), GPT-4: 10.8 (#309)

BenchmarkCodellama 34b InstructGPT-4
LMArena Math10561269
OTIS Mock AIME 2024-2025—1.1%
MATH Level 5—23%
GSM8K—92%

Knowledge Not comparable

Codellama 34b Instruct: —, GPT-4: 18.4 (#282)

BenchmarkCodellama 34b InstructGPT-4
GPQA Diamond—35.7%
LMArena Expert—1211
MMLU—86.4%
TriviaQA—84.8%

Multilingual GPT-4 leads

Codellama 34b Instruct: 25.8 (#284), GPT-4: 40.6 (#215)

BenchmarkCodellama 34b InstructGPT-4
LMArena Non-English10111246
LMArena Chinese9761242
LMArena French—1283
LMArena German—1251
LMArena Japanese—1209
LMArena Korean—1184
LMArena Russian—1251
LMArena Spanish—1261

Instruction Following GPT-4 leads

Codellama 34b Instruct: 52.2 (#291), GPT-4: 65.3 (#222)

BenchmarkCodellama 34b InstructGPT-4
LMArena Instruction Following10281241

Long Context GPT-4 leads

Codellama 34b Instruct: 30.9 (#284), GPT-4: 37.7 (#212)

BenchmarkCodellama 34b InstructGPT-4
LMArena Longer Query10131244

Writing & Preference GPT-4 leads

Codellama 34b Instruct: 28.2 (#297), GPT-4: 34.9 (#268)

BenchmarkCodellama 34b InstructGPT-4
LMArena Text10661263
LMArena Creative Writing10321244
LMArena Multi-Turn10151257
EQ-Bench Creative Writing—752

Frequently asked questions

Is Codellama 34b Instruct better than GPT-4?

Codellama 34b Instruct is the stronger model overall, scoring 30.8 to 29.1 on the Noometry Index.

Is Codellama 34b Instruct or GPT-4 better for coding?

GPT-4 scores higher on coding benchmarks: 31.6 versus 28.5 in the Noometry coding category.

How many benchmarks do Codellama 34b Instruct and GPT-4 share?

13 benchmarks have published results for both models. Codellama 34b Instruct has 14 scored results on Noometry and GPT-4 has 38.

Related comparisons

Go deeper