Noometry

Model comparison

Deepseek Coder v2 vs Phi 3 Mini 4k Instruct

Deepseek Coder v2 is the stronger model overall, scoring 35.3 to 27.4 on the Noometry Index.

Last verified . 21 shared benchmarks.

Deepseek Coder v2 DeepSeek

35.3

Rank #217 Confirmed

Phi 3 Mini 4k Instruct Microsoft

27.4

Rank #324 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Deepseek Coder v2 scores higher in 8 categories and Phi 3 Mini 4k Instruct in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in instruction following, where Deepseek Coder v2 leads 61.9 to 48.0.

Side by side

Deepseek Coder v2Phi 3 Mini 4k Instruct
ProviderDeepSeekMicrosoft
Noometry Index35.327.4
Released2024-06-172024-04-23
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked2435

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Deepseek Coder v2 leads

Deepseek Coder v2: 37.5 (#178), Phi 3 Mini 4k Instruct: 26.1 (#319)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Coding12511093
HumanEval+82.3%59.1%
MBPP+75.1%54.2%
BigCodeBench Instruct48.2%—
LiveBench Coding—15.5%
BigCodeBench Complete59.7%—

Reasoning Deepseek Coder v2 leads

Deepseek Coder v2: 22.6 (#175), Phi 3 Mini 4k Instruct: 13.6 (#326)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Hard Prompts12071072
WinoGrande83.7%70.8%
Chess Puzzles—0%
LiveBench Reasoning—26.8%
LiveBench Data Analysis—34.7%
Adversarial NLI—52.8%
BIG-Bench Hard—71.7%
HellaSwag—76.7%
LiveBench—22.4%

Math Deepseek Coder v2 leads

Deepseek Coder v2: 32.3 (#188), Phi 3 Mini 4k Instruct: 24.5 (#258)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Math12411111
LiveBench Math—15.7%
GSM8K94.5%—

Knowledge Deepseek Coder v2 leads

Deepseek Coder v2: 31.7 (#209), Phi 3 Mini 4k Instruct: 27.9 (#241)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Expert11811045
ARC (AI2) Challenge64.3%84.9%
MMLU—68.8%
OpenBookQA—88%
TriviaQA—64%

Multilingual Deepseek Coder v2 leads

Deepseek Coder v2: 36.4 (#235), Phi 3 Mini 4k Instruct: 26.5 (#275)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Non-English11821021
LMArena Chinese12011021
LMArena French11841075
LMArena German11641044
LMArena Japanese1126935
LMArena Korean1104905
LMArena Russian11881022
LMArena Spanish11531084

Instruction Following Deepseek Coder v2 leads

Deepseek Coder v2: 61.9 (#237), Phi 3 Mini 4k Instruct: 48.0 (#298)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Instruction Following11801053
LiveBench Instruction Following—39.1%

Long Context Deepseek Coder v2 leads

Deepseek Coder v2: 37.0 (#219), Phi 3 Mini 4k Instruct: 31.7 (#271)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Longer Query12191044

Writing & Preference Deepseek Coder v2 leads

Deepseek Coder v2: 38.4 (#249), Phi 3 Mini 4k Instruct: 27.7 (#295)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Text11911073
LMArena Creative Writing11201037
LMArena Multi-Turn11761018
LiveBench Language—9.2%

Frequently asked questions

Is Deepseek Coder v2 better than Phi 3 Mini 4k Instruct?

Deepseek Coder v2 is the stronger model overall, scoring 35.3 to 27.4 on the Noometry Index.

Is Deepseek Coder v2 or Phi 3 Mini 4k Instruct better for coding?

Deepseek Coder v2 scores higher on coding benchmarks: 37.5 versus 26.1 in the Noometry coding category.

How many benchmarks do Deepseek Coder v2 and Phi 3 Mini 4k Instruct share?

21 benchmarks have published results for both models. Deepseek Coder v2 has 24 scored results on Noometry and Phi 3 Mini 4k Instruct has 35.

Related comparisons

Go deeper