Noometry

Model comparison

Codellama 34b Instruct vs GPT-3.5-turbo

Codellama 34b Instruct is the stronger model overall, scoring 30.2 to 23.1 on the Noometry Index.

Last verified . 14 shared benchmarks.

Codellama 34b Instruct Meta

30.2

Rank #285 Confirmed

GPT-3.5-turbo OpenAI

23.1

Rank #345 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Codellama 34b Instruct scores higher in 4 categories and GPT-3.5-turbo in 3 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Codellama 34b Instruct leads 28.5 to 6.0.
  • The biggest single-benchmark swing is BigCodeBench Complete: 37.1% for Codellama 34b Instruct and 50.6% for GPT-3.5-turbo.
  • Codellama 34b Instruct has downloadable open weights; the other is API-only.

Side by side

Codellama 34b InstructGPT-3.5-turbo
ProviderMetaOpenAI
Noometry Index30.223.1
Released—2023-03-01
WeightsOpenProprietary
Context window—16K
Max output—4K
Input $ / M tokens—$0.50
Output $ / M tokens—$1.50
Results tracked1444

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Codellama 34b Instruct leads

Codellama 34b Instruct: 28.0 (#310), GPT-3.5-turbo: 23.5 (#326)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
BigCodeBench Instruct29%39.1%
LMArena Coding10461137
BigCodeBench Complete37.1%50.6%
HumanEval+43.9%70.7%
MBPP+56.3%69.7%
WeirdML—3.5%

Agentic & Tool Use Not comparable

Codellama 34b Instruct: —, GPT-3.5-turbo: —

BenchmarkCodellama 34b InstructGPT-3.5-turbo
METR Time Horizons—21.5%

Reasoning Codellama 34b Instruct leads

Codellama 34b Instruct: 18.8 (#250), GPT-3.5-turbo: 13.3 (#328)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
LMArena Hard Prompts10321108
Chess Puzzles—0%
Mystery Game Puzzles—3%
DTBench—48.5%
LMCA—9.7%
Adversarial NLI—58.1%
BIG-Bench Hard—61.6%
CommonsenseQA 2.0—57%
Epoch Capabilities Index—118.48
ForecastBench—50.4
WinoGrande—81.6%

Math Codellama 34b Instruct leads

Codellama 34b Instruct: 28.5 (#227), GPT-3.5-turbo: 6.0 (#321)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
LMArena Math10561142
FrontierMath (Tiers 1-3)—0%
OTIS Mock AIME 2024-2025—2.2%
MATH Level 5—15.9%
GSM8K—57.8%

Knowledge Not comparable

Codellama 34b Instruct: —, GPT-3.5-turbo: 10.0 (#298)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
GPQA Diamond—28%
LMArena Expert—1070
ARC (AI2) Challenge—87.4%
BoolQ—87%
MMLU—71.4%
OpenBookQA—86%
TriviaQA—85.8%

Multilingual GPT-3.5-turbo leads

Codellama 34b Instruct: 25.9 (#279), GPT-3.5-turbo: 31.6 (#253)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
LMArena Non-English10111108
LMArena Chinese9761075
LMArena French—1117
LMArena German—1090
LMArena Japanese—1043
LMArena Korean—1019
LMArena Russian—1123
LMArena Spanish—1120

Instruction Following GPT-3.5-turbo leads

Codellama 34b Instruct: 52.4 (#286), GPT-3.5-turbo: 58.1 (#257)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
LMArena Instruction Following10281119

Long Context GPT-3.5-turbo leads

Codellama 34b Instruct: 30.8 (#279), GPT-3.5-turbo: 34.0 (#249)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
LMArena Longer Query10141121

Writing & Preference Codellama 34b Instruct leads

Codellama 34b Instruct: 28.4 (#292), GPT-3.5-turbo: 25.4 (#300)

BenchmarkCodellama 34b InstructGPT-3.5-turbo
LMArena Text10661125
LMArena Creative Writing10321092
LMArena Multi-Turn10151117
EQ-Bench Creative Writing—451

Frequently asked questions

Is Codellama 34b Instruct better than GPT-3.5-turbo?

Codellama 34b Instruct is the stronger model overall, scoring 30.2 to 23.1 on the Noometry Index.

Is Codellama 34b Instruct or GPT-3.5-turbo better for coding?

Codellama 34b Instruct scores higher on coding benchmarks: 28.0 versus 23.5 in the Noometry coding category.

How many benchmarks do Codellama 34b Instruct and GPT-3.5-turbo share?

14 benchmarks have published results for both models. Codellama 34b Instruct has 14 scored results on Noometry and GPT-3.5-turbo has 44.

Related comparisons

Go deeper