Noometry

Model comparison

Grok-2 (Dec 2024) vs Mistral Nemo

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.4 to 25.9 on the Noometry Index.

Last verified . 4 shared benchmarks.

Grok-2 (Dec 2024) xAI

33.4

Rank #233 Confirmed

Mistral Nemo Mistral AI

25.9

Rank #332 Confirmed

Summary

  • They share 4 benchmarks with published results for both. Grok-2 (Dec 2024) scores higher in 2 categories and Mistral Nemo in 2 categories; 4 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Grok-2 (Dec 2024) leads 48.8 to 28.6.
  • The biggest single-benchmark swing is MATH Level 5: 63.5% for Grok-2 (Dec 2024) and 10.8% for Mistral Nemo.
  • Mistral Nemo has downloadable open weights; the other is API-only.

Side by side

Grok-2 (Dec 2024)Mistral Nemo
ProviderxAIMistral AI
Noometry Index33.425.9
Released2024-08-132024-07-01
WeightsProprietaryOpen
Context window—128K
Max output—128K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.15
Results tracked3410

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Grok-2 (Dec 2024): 32.9 (#254), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
WeirdML22.2%—
LiveBench Coding46.4%—
LMArena Coding1288—

Agentic & Tool Use Not comparable

Grok-2 (Dec 2024): —, Mistral Nemo: 23.3 (#118)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
Berkeley Function Calling Leaderboard—27.6%
BALROG—17.6%

Reasoning Mistral Nemo leads

Grok-2 (Dec 2024): 16.4 (#294), Mistral Nemo: 19.9 (#230)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
DTBench65.2%48.6%
Epoch Capabilities Index130.48118.63
SimpleBench22.7%—
LiveBench Reasoning54.8%—
LMArena Hard Prompts1272—
LiveBench Data Analysis54.5%—
LiveBench54.3%—
PIQA—83.5%

Math Mistral Nemo leads

Grok-2 (Dec 2024): 19.6 (#278), Mistral Nemo: 23.5 (#263)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
MATH Level 563.5%10.8%
OTIS Mock AIME 2024-202511.5%—
LiveBench Math54.9%—
LMArena Math1283—
FrontierMath (Feb 2025 set)0.7%—
GSM8K—84.2%

Knowledge Grok-2 (Dec 2024) leads

Grok-2 (Dec 2024): 29.8 (#225), Mistral Nemo: 12.3 (#293)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
GPQA Diamond53.8%29.9%
Confabulations20.1%—
LMArena Expert1254—
BoolQ—82.5%

Multilingual Not comparable

Grok-2 (Dec 2024): 43.3 (#183), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LMArena Non-English1282—
LMArena Chinese1289—
LMArena French1317—
LMArena German1287—
LMArena Japanese1244—
LMArena Korean1237—
LMArena Russian1286—
LMArena Spanish1281—

Instruction Following Not comparable

Grok-2 (Dec 2024): 67.1 (#197), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LiveBench Instruction Following69.6%—
LMArena Instruction Following1270—

Long Context Not comparable

Grok-2 (Dec 2024): 38.7 (#186), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LMArena Longer Query1276—

Writing & Preference Grok-2 (Dec 2024) leads

Grok-2 (Dec 2024): 48.8 (#192), Mistral Nemo: 28.6 (#291)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LMArena Text1305—
LMArena Creative Writing1284—
Short-Story Creative Writing63.6%—
EQ-Bench Creative Writing—881
LMArena Multi-Turn1290—
LiveBench Language45.6%—

Frequently asked questions

Is Grok-2 (Dec 2024) better than Mistral Nemo?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.4 to 25.9 on the Noometry Index.

How many benchmarks do Grok-2 (Dec 2024) and Mistral Nemo share?

4 benchmarks have published results for both models. Grok-2 (Dec 2024) has 34 scored results on Noometry and Mistral Nemo has 10.

Related comparisons

Go deeper