Noometry

Model comparison

Gemma 3 4B vs Phi-4

Phi-4 is the stronger model overall, scoring 30.9 to 27.9 on the Noometry Index. Gemma 3 4B costs 1.8× less per token, which makes it the better buy when Phi-4's lead doesn't matter for your workload.

Last verified . 18 shared benchmarks.

Gemma 3 4B Google

27.9

Rank #320 Confirmed

Phi-4 Microsoft

30.9

Rank #269 Confirmed

Summary

  • They share 18 benchmarks with published results for both. Gemma 3 4B scores higher in 5 categories and Phi-4 in 4 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Phi-4 leads 32.5 to 11.8.
  • The biggest single-benchmark swing is GPQA Diamond: 23.2% for Gemma 3 4B and 56.1% for Phi-4.
  • Gemma 3 4B is cheaper at $0.04 / $0.08 per million input/output tokens, against $0.07 / $0.14 for Phi-4.
  • Gemma 3 4B accepts more context: 131K tokens versus 128K.

Side by side

Gemma 3 4BPhi-4
ProviderGoogleMicrosoft
Noometry Index27.930.9
Released2025-03-122024-12-11
WeightsOpenOpen
Context window131K128K
Max output4K4K
Input $ / M tokens$0.04$0.07
Output $ / M tokens$0.08$0.14
Results tracked2237

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 3 4B leads

Gemma 3 4B: 35.3 (#214), Phi-4: 33.8 (#237)

BenchmarkGemma 3 4BPhi-4
LMArena Coding12301231
BigCodeBench Instruct—45.5%
LiveBench Coding—30.7%
BigCodeBench Complete—55.4%

Agentic & Tool Use Phi-4 leads

Gemma 3 4B: 20.7 (#134), Phi-4: 22.6 (#120)

BenchmarkGemma 3 4BPhi-4
Berkeley Function Calling Leaderboard19.6%28.8%
BALROG—11.6%

Reasoning Phi-4 leads

Gemma 3 4B: 12.7 (#330), Phi-4: 17.0 (#287)

BenchmarkGemma 3 4BPhi-4
Chess Puzzles0%1%
LMArena Hard Prompts12531220
Epoch Capabilities Index116.02130.43
Kagi LLM Benchmark25.2%—
LiveBench Reasoning—47.8%
DTBench50.9%—
LiveBench Data Analysis—45.2%
LMCA2.8%—
LiveBench—41.6%

Math Phi-4 leads

Gemma 3 4B: 15.8 (#286), Phi-4: 19.6 (#279)

BenchmarkGemma 3 4BPhi-4
OTIS Mock AIME 2024-20257.5%13.8%
LMArena Math12391246
LiveBench Math—42%
MATH Level 5—64.9%

Knowledge Phi-4 leads

Gemma 3 4B: 11.8 (#294), Phi-4: 32.5 (#204)

BenchmarkGemma 3 4BPhi-4
GPQA Diamond23.2%56.1%
Vectara Hallucination Rate6.4%3.7%
LMArena Expert12231203
Confabulations—29.4%
MMLU—84.8%

Multilingual Gemma 3 4B leads

Gemma 3 4B: 42.7 (#189), Phi-4: 37.4 (#232)

BenchmarkGemma 3 4BPhi-4
LMArena Non-English12731197
LMArena German12811222
LMArena Russian12941209
LMArena Chinese—1211
LMArena French—1223
LMArena Japanese—1158
LMArena Korean—1151
LMArena Spanish—1234

Instruction Following Gemma 3 4B leads

Gemma 3 4B: 65.4 (#219), Phi-4: 60.6 (#246)

BenchmarkGemma 3 4BPhi-4
LMArena Instruction Following12391201
LiveBench Instruction Following—58.4%

Long Context Gemma 3 4B leads

Gemma 3 4B: 38.6 (#189), Phi-4: 36.9 (#221)

BenchmarkGemma 3 4BPhi-4
LMArena Longer Query12741217

Writing & Preference Gemma 3 4B leads

Gemma 3 4B: 42.2 (#234), Phi-4: 40.7 (#239)

BenchmarkGemma 3 4BPhi-4
LMArena Text12911217
LMArena Creative Writing12701182
LMArena Multi-Turn12551206
Short-Story Creative Writing—62.6%
EQ-Bench Creative Writing1068—
LiveBench Language—25.6%

Frequently asked questions

Is Gemma 3 4B better than Phi-4?

Phi-4 is the stronger model overall, scoring 30.9 to 27.9 on the Noometry Index. Gemma 3 4B costs 1.8× less per token, which makes it the better buy when Phi-4's lead doesn't matter for your workload.

Which is cheaper, Gemma 3 4B or Phi-4?

Gemma 3 4B is cheaper. It lists at $0.04 per million input tokens and $0.08 per million output tokens; Phi-4 lists at $0.07 and $0.14.

Is Gemma 3 4B or Phi-4 better for coding?

Gemma 3 4B scores higher on coding benchmarks: 35.3 versus 33.8 in the Noometry coding category.

Which has the bigger context window?

Gemma 3 4B does, with 131K tokens against 128K.

How many benchmarks do Gemma 3 4B and Phi-4 share?

18 benchmarks have published results for both models. Gemma 3 4B has 22 scored results on Noometry and Phi-4 has 37.

Related comparisons

Go deeper