Noometry

Model comparison

Deepseek Coder v2 vs Llama 3.1-70B

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 29.6 on the Noometry Index.

Last verified . 19 shared benchmarks.

Deepseek Coder v2 DeepSeek

35.9

Rank #220 Confirmed

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Summary

  • They share 19 benchmarks with published results for both. Deepseek Coder v2 scores higher in 5 categories and Llama 3.1-70B in 3 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Deepseek Coder v2 leads 34.9 to 13.5.

Side by side

Deepseek Coder v2Llama 3.1-70B
ProviderDeepSeekMeta
Noometry Index35.929.6
Released2024-06-172024-07-23
WeightsOpenOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.40
Output $ / M tokens—$0.40
Results tracked2435

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Deepseek Coder v2 leads

Deepseek Coder v2: 38.1 (#183), Llama 3.1-70B: 30.3 (#296)

BenchmarkDeepseek Coder v2Llama 3.1-70B
BigCodeBench Instruct48.2%46.1%
LMArena Coding12511260
BigCodeBench Complete59.7%54.8%
WeirdML—9%
HumanEval+82.3%—
MBPP+75.1%—

Agentic & Tool Use Not comparable

Deepseek Coder v2: —, Llama 3.1-70B: 25.1 (#112)

BenchmarkDeepseek Coder v2Llama 3.1-70B
TheAgentCompany—6.9%
BALROG—27.9%

Reasoning Deepseek Coder v2 leads

Deepseek Coder v2: 23.6 (#176), Llama 3.1-70B: 21.6 (#220)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Hard Prompts12071241
DTBench—60%
LMCA—14.8%
Epoch Capabilities Index—125.92
WinoGrande83.7%—

Math Deepseek Coder v2 leads

Deepseek Coder v2: 34.9 (#190), Llama 3.1-70B: 13.5 (#304)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Math12411252
OTIS Mock AIME 2024-2025—3.6%
Omni-MATH—21%
MATH Level 5—36.7%
GSM8K94.5%—

Knowledge Deepseek Coder v2 leads

Deepseek Coder v2: 32.3 (#212), Llama 3.1-70B: 24.2 (#269)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Expert11811209
GPQA Diamond—44.2%
MMLU-Pro—65.3%
GPQA (HELM)—42.6%
ARC (AI2) Challenge64.3%—
MMLU—80.1%

Multilingual Llama 3.1-70B leads

Deepseek Coder v2: 36.3 (#240), Llama 3.1-70B: 38.8 (#225)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Non-English11821219
LMArena Chinese12011215
LMArena French11851261
LMArena German11641222
LMArena Japanese11261132
LMArena Korean11041140
LMArena Russian11881234
LMArena Spanish11531253

Instruction Following Llama 3.1-70B leads

Deepseek Coder v2: 61.7 (#242), Llama 3.1-70B: 65.3 (#223)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Instruction Following11801231
IFEval—82.1%

Long Context Too close to call

Deepseek Coder v2: 37.0 (#224), Llama 3.1-70B: 37.6 (#214)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Longer Query12191241

Writing & Preference Deepseek Coder v2 leads

Deepseek Coder v2: 38.2 (#253), Llama 3.1-70B: 35.4 (#267)

BenchmarkDeepseek Coder v2Llama 3.1-70B
LMArena Text11911261
LMArena Creative Writing11201232
LMArena Multi-Turn11771256
EQ-Bench Creative Writing—784
WildBench—75.8%

Frequently asked questions

Is Deepseek Coder v2 better than Llama 3.1-70B?

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 29.6 on the Noometry Index.

Is Deepseek Coder v2 or Llama 3.1-70B better for coding?

Deepseek Coder v2 scores higher on coding benchmarks: 38.1 versus 30.3 in the Noometry coding category.

How many benchmarks do Deepseek Coder v2 and Llama 3.1-70B share?

19 benchmarks have published results for both models. Deepseek Coder v2 has 24 scored results on Noometry and Llama 3.1-70B has 35.

Related comparisons

Go deeper