Model comparison

Gemma 7B vs GPT-4o mini

Gemma 7B is the stronger model overall, scoring 29.5 to 25.4 on the Noometry Index.

Last verified October 10, 2026. 20 shared benchmarks.

Gemma 7B

Google

29.5

Rank #295 Confirmed

GPT-4o mini OpenAI

25.4

Rank #337 Confirmed

Summary

They share 20 benchmarks with published results for both. Gemma 7B scores higher in 4 categories and GPT-4o mini in 4 categories; 8 gaps are clear of the uncertainty.
The widest gap is in math, where Gemma 7B leads 28.7 to 9.9.
Gemma 7B has downloadable open weights; the other is API-only.

Side by side

Gemma 7B and GPT-4o mini specifications
	Gemma 7B	GPT-4o mini
Provider	Google	OpenAI
Noometry Index	29.5	25.4
Released	2024-02-21	2024-07-18
Weights	Open	Proprietary
Context window	—	128K
Max output	—	16K
Input $ / M tokens	—	$0.15
Output $ / M tokens	—	$0.60
Results tracked	27	60

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 7B leads

Gemma 7B: 29.9 (#290), GPT-4o mini: 21.6 (#330)

Coding benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Coding	1048	1290
HumanEval+	28.7%	83.5%
MBPP+	43.4%	72.2%
Aider Polyglot	—	3.6%
WeirdML	—	11.8%
BigCodeBench Instruct	—	46.1%
LiveBench Coding	—	43.1%
BigCodeBench Complete	—	57.4%

Agentic & Tool Use Not comparable

Gemma 7B: —, GPT-4o mini: 27.2 (#95)

Agentic & Tool Use benchmarks
Benchmark	Gemma 7B	GPT-4o mini
BALROG	—	17.4%

Reasoning Gemma 7B leads

Gemma 7B: 19.0 (#247), GPT-4o mini: 8.7 (#343)

Reasoning benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Hard Prompts	1042	1267
Epoch Capabilities Index	111.74	126.56
PIQA	81.2%	88.7%
ARC-AGI-2	—	0%
SimpleBench	—	10.7%
Kagi LLM Benchmark	—	28.8%
Chess Puzzles	—	0%
LiveBench Reasoning	—	32.8%
Mystery Game Puzzles	—	12%
DTBench	—	54.4%
LiveBench Data Analysis	—	50%
LMCA	—	10.4%
Adversarial NLI	48.7%	—
BIG-Bench Hard	55.1%	—
HellaSwag	82.2%	—
LiveBench	—	41.3%
WinoGrande	79%	—

Math Gemma 7B leads

Gemma 7B: 28.7 (#225), GPT-4o mini: 9.9 (#307)

Math benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Math	1066	1267
GSM8K	46.4%	91.3%
FrontierMath (Tiers 1-3)	—	0.7%
OTIS Mock AIME 2024-2025	—	6.9%
Omni-MATH	—	28%
LiveBench Math	—	36.3%
MATH Level 5	—	52.6%

Knowledge Gemma 7B leads

Gemma 7B: 26.8 (#248), GPT-4o mini: 17.7 (#279)

Knowledge benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Expert	1002	1235
BoolQ	83.2%	88.7%
MMLU	66.1%	81.8%
GPQA Diamond	—	37.7%
SimpleQA Verified	—	8.3%
MMLU-Pro	—	60.3%
Confabulations	—	37.2%
GPQA (HELM)	—	36.8%
ARC (AI2) Challenge	78.3%	—
OpenBookQA	78.6%	—
TriviaQA	72.3%	—

Multimodal Not comparable

Gemma 7B: —, GPT-4o mini: 25.3 (#117)

Multimodal benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Vision	—	1066
Video-MME	—	64.8%
GeoBench	—	64%
VPCT	—	34%

Multilingual GPT-4o mini leads

Gemma 7B: 25.3 (#282), GPT-4o mini: 42.2 (#194)

Multilingual benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Non-English	999	1266
LMArena Chinese	1035	1264
LMArena French	1024	1296
LMArena Russian	993	1275
LMArena German	—	1272
LMArena Japanese	—	1216
LMArena Korean	—	1195
LMArena Spanish	—	1276

Instruction Following GPT-4o mini leads

Gemma 7B: 51.7 (#290), GPT-4o mini: 62.1 (#234)

Instruction Following benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Instruction Following	1017	1259
LiveBench Instruction Following	—	56.8%
IFEval	—	78.2%

Long Context GPT-4o mini leads

Gemma 7B: 31.1 (#277), GPT-4o mini: 39.1 (#181)

Long Context benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Longer Query	1022	1289

Writing & Preference GPT-4o mini leads

Gemma 7B: 27.2 (#297), GPT-4o mini: 39.7 (#243)

Writing & Preference benchmarks
Benchmark	Gemma 7B	GPT-4o mini
LMArena Text	1056	1286
LMArena Creative Writing	1024	1268
LMArena Multi-Turn	963	1285
Short-Story Creative Writing	—	67.2%
EQ-Bench Creative Writing	—	873
WildBench	—	79.1%
LiveBench Language	—	28.6%

Frequently asked questions

Is Gemma 7B better than GPT-4o mini?