Noometry

Model comparison

Llama 3.1-70B vs phi-3-medium 14B

Llama 3.1-70B and phi-3-medium 14B score almost the same on the Noometry Index (29.6 vs 29.7), so choose on price, context window or the category you care about most.

Last verified . 6 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

phi-3-medium 14B Microsoft

29.7

Rank #306 Reported

Summary

  • They share 6 benchmarks with published results for both. Llama 3.1-70B scores higher in 1 category and phi-3-medium 14B in 2 categories; 3 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Llama 3.1-70B leads 24.2 to 9.1.
  • The biggest single-benchmark swing is MATH Level 5: 36.7% for Llama 3.1-70B and 17.6% for phi-3-medium 14B.

Side by side

Llama 3.1-70Bphi-3-medium 14B
ProviderMetaMicrosoft
Noometry Index29.629.7
Released2024-07-232024-04-23
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.40—
Output $ / M tokens$0.40—
Results tracked3513

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding phi-3-medium 14B leads

Llama 3.1-70B: 30.3 (#296), phi-3-medium 14B: 36.8 (#201)

BenchmarkLlama 3.1-70Bphi-3-medium 14B
BigCodeBench Instruct46.1%37.6%
BigCodeBench Complete54.8%48.7%
WeirdML9%—
LMArena Coding1260—

Agentic & Tool Use Not comparable

Llama 3.1-70B: 25.1 (#112), phi-3-medium 14B: —

BenchmarkLlama 3.1-70Bphi-3-medium 14B
TheAgentCompany6.9%—
BALROG27.9%—

Reasoning Not comparable

Llama 3.1-70B: 21.6 (#220), phi-3-medium 14B: —

BenchmarkLlama 3.1-70Bphi-3-medium 14B
Epoch Capabilities Index125.92121.23
LMArena Hard Prompts1241—
DTBench60%—
LMCA14.8%—
Adversarial NLI—55.8%
BIG-Bench Hard—81.4%
HellaSwag—82.4%
WinoGrande—81.5%

Math phi-3-medium 14B leads

Llama 3.1-70B: 13.5 (#304), phi-3-medium 14B: 27.3 (#250)

BenchmarkLlama 3.1-70Bphi-3-medium 14B
MATH Level 536.7%17.6%
OTIS Mock AIME 2024-20253.6%—
Omni-MATH21%—
LMArena Math1252—

Knowledge Llama 3.1-70B leads

Llama 3.1-70B: 24.2 (#269), phi-3-medium 14B: 9.1 (#306)

BenchmarkLlama 3.1-70Bphi-3-medium 14B
GPQA Diamond44.2%27.6%
MMLU80.1%78%
MMLU-Pro65.3%—
GPQA (HELM)42.6%—
LMArena Expert1209—
ARC (AI2) Challenge—91.6%
OpenBookQA—87.4%
TriviaQA—73.9%

Multilingual Not comparable

Llama 3.1-70B: 38.8 (#225), phi-3-medium 14B: —

BenchmarkLlama 3.1-70Bphi-3-medium 14B
LMArena Non-English1219—
LMArena Chinese1215—
LMArena French1261—
LMArena German1222—
LMArena Japanese1132—
LMArena Korean1140—
LMArena Russian1234—
LMArena Spanish1253—

Instruction Following Not comparable

Llama 3.1-70B: 65.3 (#223), phi-3-medium 14B: —

BenchmarkLlama 3.1-70Bphi-3-medium 14B
IFEval82.1%—
LMArena Instruction Following1231—

Long Context Not comparable

Llama 3.1-70B: 37.6 (#214), phi-3-medium 14B: —

BenchmarkLlama 3.1-70Bphi-3-medium 14B
LMArena Longer Query1241—

Writing & Preference Not comparable

Llama 3.1-70B: 35.4 (#267), phi-3-medium 14B: —

BenchmarkLlama 3.1-70Bphi-3-medium 14B
LMArena Text1261—
LMArena Creative Writing1232—
EQ-Bench Creative Writing784—
WildBench75.8%—
LMArena Multi-Turn1256—

Frequently asked questions

Is Llama 3.1-70B better than phi-3-medium 14B?

Llama 3.1-70B and phi-3-medium 14B score almost the same on the Noometry Index (29.6 vs 29.7), so choose on price, context window or the category you care about most.

Is Llama 3.1-70B or phi-3-medium 14B better for coding?

phi-3-medium 14B scores higher on coding benchmarks: 36.8 versus 30.3 in the Noometry coding category.

How many benchmarks do Llama 3.1-70B and phi-3-medium 14B share?

6 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and phi-3-medium 14B has 13.

Related comparisons

Go deeper