Noometry

Model comparison

Deepseek Coder v2 vs Mixtral 8x22B

Deepseek Coder v2 is the stronger model overall, scoring 35.3 to 26.7 on the Noometry Index.

Last verified . 21 shared benchmarks.

Deepseek Coder v2 DeepSeek

35.3

Rank #217 Confirmed

Mixtral 8x22B Mistral AI

26.7

Rank #330 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Deepseek Coder v2 scores higher in 8 categories and Mixtral 8x22B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Deepseek Coder v2 leads 31.7 to 15.1.
  • The biggest single-benchmark swing is BigCodeBench Complete: 59.7% for Deepseek Coder v2 and 50.2% for Mixtral 8x22B.

Side by side

Deepseek Coder v2Mixtral 8x22B
ProviderDeepSeekMistral AI
Noometry Index35.326.7
Released2024-06-172024-04-17
WeightsOpenOpen
Context window—64K
Max output—64K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked2434

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Deepseek Coder v2 leads

Deepseek Coder v2: 37.5 (#178), Mixtral 8x22B: 23.8 (#324)

BenchmarkDeepseek Coder v2Mixtral 8x22B
BigCodeBench Instruct48.2%40.6%
LMArena Coding12511166
BigCodeBench Complete59.7%50.2%
HumanEval+82.3%72%
MBPP+75.1%64.3%
WeirdML—3.2%

Agentic & Tool Use Not comparable

Deepseek Coder v2: —, Mixtral 8x22B: 22.8 (#119)

BenchmarkDeepseek Coder v2Mixtral 8x22B
Cybench—7.5%

Reasoning Deepseek Coder v2 leads

Deepseek Coder v2: 22.6 (#175), Mixtral 8x22B: 19.1 (#246)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Hard Prompts12071150
DTBench—55.1%
Epoch Capabilities Index—122
ForecastBench—56.3
WinoGrande83.7%—

Math Deepseek Coder v2 leads

Deepseek Coder v2: 32.3 (#188), Mixtral 8x22B: 21.2 (#271)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Math12411184
Omni-MATH—16.3%
MATH Level 5—24.2%
GSM8K94.5%—

Knowledge Deepseek Coder v2 leads

Deepseek Coder v2: 31.7 (#209), Mixtral 8x22B: 15.1 (#288)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Expert11811113
GPQA Diamond—34.1%
MMLU-Pro—46%
GPQA (HELM)—33.4%
ARC (AI2) Challenge64.3%—
MMLU—77.8%

Multilingual Deepseek Coder v2 leads

Deepseek Coder v2: 36.4 (#235), Mixtral 8x22B: 32.9 (#250)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Non-English11821128
LMArena Chinese12011116
LMArena French11841165
LMArena German11641141
LMArena Japanese11261037
LMArena Korean11041056
LMArena Russian11881158
LMArena Spanish11531150

Instruction Following Deepseek Coder v2 leads

Deepseek Coder v2: 61.9 (#237), Mixtral 8x22B: 57.9 (#261)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Instruction Following11801147
IFEval—72.4%

Long Context Deepseek Coder v2 leads

Deepseek Coder v2: 37.0 (#219), Mixtral 8x22B: 34.6 (#242)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Longer Query12191144

Writing & Preference Deepseek Coder v2 leads

Deepseek Coder v2: 38.4 (#249), Mixtral 8x22B: 37.1 (#257)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Text11911162
LMArena Creative Writing11201141
LMArena Multi-Turn11761130
WildBench—71.1%

Frequently asked questions

Is Deepseek Coder v2 better than Mixtral 8x22B?

Deepseek Coder v2 is the stronger model overall, scoring 35.3 to 26.7 on the Noometry Index.

Is Deepseek Coder v2 or Mixtral 8x22B better for coding?

Deepseek Coder v2 scores higher on coding benchmarks: 37.5 versus 23.8 in the Noometry coding category.

How many benchmarks do Deepseek Coder v2 and Mixtral 8x22B share?

21 benchmarks have published results for both models. Deepseek Coder v2 has 24 scored results on Noometry and Mixtral 8x22B has 34.

Related comparisons

Go deeper