Noometry

Model comparison

Gemma 1.1 7b IT vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.1 to 30.7 on the Noometry Index.

Last verified . 17 shared benchmarks.

Gemma 1.1 7b IT Google

30.7

Rank #275 Confirmed

Mistral Small Mistral AI

33.1

Rank #238 Confirmed

Summary

  • They share 17 benchmarks with published results for both. Gemma 1.1 7b IT scores higher in 2 categories and Mistral Small in 6 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.6 to 30.6.

Side by side

Gemma 1.1 7b ITMistral Small
ProviderGoogleMistral AI
Noometry Index30.733.1
Released—2024-02-26
WeightsOpenOpen
Context window—256K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked1939

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small leads

Gemma 1.1 7b IT: 31.0 (#279), Mistral Small: 33.5 (#242)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Coding10841363
SciCode—26.5%
BigCodeBench Instruct—36.1%
LiveBench Coding—36.2%
BigCodeBench Complete—46.6%
ALE-Bench—497.62
HumanEval+35.4%—
MBPP+45%—

Agentic & Tool Use Not comparable

Gemma 1.1 7b IT: —, Mistral Small: 27.9 (#87)

BenchmarkGemma 1.1 7b ITMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Too close to call

Gemma 1.1 7b IT: 19.6 (#235), Mistral Small: 19.3 (#238)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Hard Prompts10711336
Kagi LLM Benchmark—37.8%
CritPt—0%
LiveBench Reasoning—44.8%
DTBench—70.9%
LiveBench Data Analysis—53.7%
LMCA—20.6%
LiveBench—44%

Math Gemma 1.1 7b IT leads

Gemma 1.1 7b IT: 29.5 (#217), Mistral Small: 15.4 (#287)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Math11071341
OTIS Mock AIME 2024-2025—5.8%
LiveBench Math—39.9%
MATH Level 5—46.8%

Knowledge Mistral Small leads

Gemma 1.1 7b IT: 27.8 (#242), Mistral Small: 30.9 (#214)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Expert10391293
GPQA Diamond—47.5%
Vectara Hallucination Rate—5.1%
MMLU—68.7%

Multimodal Not comparable

Gemma 1.1 7b IT: —, Mistral Small: 32.9 (#91)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Vision—1142

Multilingual Mistral Small leads

Gemma 1.1 7b IT: 28.3 (#268), Mistral Small: 45.7 (#164)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Non-English10521315
LMArena Chinese10601342
LMArena French10641337
LMArena German10541341
LMArena Japanese9711275
LMArena Korean9881260
LMArena Russian10461324
LMArena Spanish10491346

Instruction Following Mistral Small leads

Gemma 1.1 7b IT: 54.3 (#278), Mistral Small: 66.6 (#204)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Instruction Following10571310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Gemma 1.1 7b IT: 32.0 (#267), Mistral Small: 40.4 (#151)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Longer Query10561327

Writing & Preference Mistral Small leads

Gemma 1.1 7b IT: 30.6 (#283), Mistral Small: 52.6 (#165)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Text10941338
LMArena Creative Writing10591304
LMArena Multi-Turn10401344
LiveBench Language—30.5%

Frequently asked questions

Is Gemma 1.1 7b IT better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.1 to 30.7 on the Noometry Index.

Is Gemma 1.1 7b IT or Mistral Small better for coding?

Mistral Small scores higher on coding benchmarks: 33.5 versus 31.0 in the Noometry coding category.

How many benchmarks do Gemma 1.1 7b IT and Mistral Small share?

17 benchmarks have published results for both models. Gemma 1.1 7b IT has 19 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper