Noometry

Model comparison

Phi-4 vs Tulu 3 (Tülu 3) 70B

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 32.6 to 30.9 on the Noometry Index.

Last verified . 14 shared benchmarks.

Phi-4 Microsoft

30.9

Rank #269 Confirmed

Tulu 3 (Tülu 3) 70B Allen Institute for AI (Ai2)

32.6

Rank #242 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Phi-4 scores higher in 2 categories and Tulu 3 (Tülu 3) 70B in 6 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Phi-4 leads 32.5 to 25.1.
  • The biggest single-benchmark swing is MATH Level 5: 64.9% for Phi-4 and 42.7% for Tulu 3 (Tülu 3) 70B.

Side by side

Phi-4Tulu 3 (Tülu 3) 70B
ProviderMicrosoftAllen Institute for AI (Ai2)
Noometry Index30.932.6
Released2024-12-112024-11-21
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.07—
Output $ / M tokens$0.14—
Results tracked3714

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Tulu 3 (Tülu 3) 70B leads

Phi-4: 33.8 (#237), Tulu 3 (Tülu 3) 70B: 35.4 (#211)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
LMArena Coding12311236
BigCodeBench Instruct45.5%—
LiveBench Coding30.7%—
BigCodeBench Complete55.4%—

Agentic & Tool Use Not comparable

Phi-4: 22.6 (#120), Tulu 3 (Tülu 3) 70B: —

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
Berkeley Function Calling Leaderboard28.8%—
BALROG11.6%—

Reasoning Tulu 3 (Tülu 3) 70B leads

Phi-4: 17.0 (#287), Tulu 3 (Tülu 3) 70B: 22.9 (#172)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
LMArena Hard Prompts12201220
Chess Puzzles1%—
LiveBench Reasoning47.8%—
LiveBench Data Analysis45.2%—
Epoch Capabilities Index130.43—
LiveBench41.6%—

Math Phi-4 leads

Phi-4: 19.6 (#279), Tulu 3 (Tülu 3) 70B: 13.3 (#297)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
OTIS Mock AIME 2024-202513.8%4.4%
LMArena Math12461242
MATH Level 564.9%42.7%
LiveBench Math42%—

Knowledge Phi-4 leads

Phi-4: 32.5 (#204), Tulu 3 (Tülu 3) 70B: 25.1 (#257)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
GPQA Diamond56.1%46.3%
Confabulations29.4%—
Vectara Hallucination Rate3.7%—
LMArena Expert1203—
MMLU84.8%—

Multilingual Tulu 3 (Tülu 3) 70B leads

Phi-4: 37.4 (#232), Tulu 3 (Tülu 3) 70B: 40.1 (#217)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
LMArena Non-English11971236
LMArena Chinese12111249
LMArena Russian12091246
LMArena French1223—
LMArena German1222—
LMArena Japanese1158—
LMArena Korean1151—
LMArena Spanish1234—

Instruction Following Tulu 3 (Tülu 3) 70B leads

Phi-4: 60.6 (#246), Tulu 3 (Tülu 3) 70B: 65.0 (#222)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
LMArena Instruction Following12011233
LiveBench Instruction Following58.4%—

Long Context Too close to call

Phi-4: 36.9 (#221), Tulu 3 (Tülu 3) 70B: 37.1 (#217)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
LMArena Longer Query12171224

Writing & Preference Tulu 3 (Tülu 3) 70B leads

Phi-4: 40.7 (#239), Tulu 3 (Tülu 3) 70B: 45.8 (#218)

BenchmarkPhi-4Tulu 3 (Tülu 3) 70B
LMArena Text12171256
LMArena Creative Writing11821231
LMArena Multi-Turn12061252
Short-Story Creative Writing62.6%—
LiveBench Language25.6%—

Frequently asked questions

Is Phi-4 better than Tulu 3 (Tülu 3) 70B?

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 32.6 to 30.9 on the Noometry Index.

Is Phi-4 or Tulu 3 (Tülu 3) 70B better for coding?

Tulu 3 (Tülu 3) 70B scores higher on coding benchmarks: 35.4 versus 33.8 in the Noometry coding category.

How many benchmarks do Phi-4 and Tulu 3 (Tülu 3) 70B share?

14 benchmarks have published results for both models. Phi-4 has 37 scored results on Noometry and Tulu 3 (Tülu 3) 70B has 14.

Related comparisons

Go deeper