Noometry

Model comparison

Mistral Nemo vs Mixtral 8x22B

Mistral Nemo and Mixtral 8x22B score almost the same on the Noometry Index (26.4 vs 27.1), so choose on price, context window or the category you care about most.

Last verified . 4 shared benchmarks.

Mistral Nemo Mistral AI

26.4

Rank #337 Confirmed

Mixtral 8x22B Mistral AI

27.1

Rank #333 Confirmed

Summary

  • They share 4 benchmarks with published results for both. Mistral Nemo scores higher in 3 categories and Mixtral 8x22B in 2 categories; 3 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mixtral 8x22B leads 36.9 to 28.5.
  • The biggest single-benchmark swing is MATH Level 5: 10.8% for Mistral Nemo and 24.2% for Mixtral 8x22B.
  • Mistral Nemo is cheaper at $0.15 / $0.15 per million input/output tokens, against $2 / $6 for Mixtral 8x22B.
  • Mistral Nemo accepts more context: 128K tokens versus 64K.

Side by side

Mistral NemoMixtral 8x22B
ProviderMistral AIMistral AI
Noometry Index26.427.1
Released2024-07-012024-04-17
WeightsOpenOpen
Context window128K64K
Max output128K64K
Input $ / M tokens$0.15$2
Output $ / M tokens$0.15$6
Results tracked1034

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Mistral Nemo: —, Mixtral 8x22B: 24.2 (#329)

BenchmarkMistral NemoMixtral 8x22B
WeirdML—3.2%
BigCodeBench Instruct—40.6%
LMArena Coding—1166
BigCodeBench Complete—50.2%
HumanEval+—72%
MBPP+—64.3%

Agentic & Tool Use Too close to call

Mistral Nemo: 23.5 (#125), Mixtral 8x22B: 23.1 (#127)

BenchmarkMistral NemoMixtral 8x22B
Berkeley Function Calling Leaderboard27.6%—
Cybench—7.5%
BALROG17.6%—

Reasoning Too close to call

Mistral Nemo: 20.7 (#232), Mixtral 8x22B: 19.9 (#248)

BenchmarkMistral NemoMixtral 8x22B
DTBench48.6%55.1%
Epoch Capabilities Index118.68122.03
LMArena Hard Prompts—1150
ForecastBench—56.3
PIQA83.5%—

Math Mistral Nemo leads

Mistral Nemo: 25.5 (#268), Mixtral 8x22B: 22.9 (#275)

BenchmarkMistral NemoMixtral 8x22B
MATH Level 510.8%24.2%
Omni-MATH—16.3%
LMArena Math—1184
GSM8K84.2%—

Knowledge Mixtral 8x22B leads

Mistral Nemo: 12.3 (#298), Mixtral 8x22B: 15.1 (#293)

BenchmarkMistral NemoMixtral 8x22B
GPQA Diamond29.9%34.1%
MMLU-Pro—46%
GPQA (HELM)—33.4%
LMArena Expert—1113
BoolQ82.5%—
MMLU—77.8%

Multilingual Not comparable

Mistral Nemo: —, Mixtral 8x22B: 32.8 (#255)

BenchmarkMistral NemoMixtral 8x22B
LMArena Non-English—1128
LMArena Chinese—1116
LMArena French—1166
LMArena German—1141
LMArena Japanese—1037
LMArena Korean—1057
LMArena Russian—1158
LMArena Spanish—1151

Instruction Following Not comparable

Mistral Nemo: —, Mixtral 8x22B: 57.7 (#266)

BenchmarkMistral NemoMixtral 8x22B
IFEval—72.4%
LMArena Instruction Following—1147

Long Context Not comparable

Mistral Nemo: —, Mixtral 8x22B: 34.7 (#247)

BenchmarkMistral NemoMixtral 8x22B
LMArena Longer Query—1144

Writing & Preference Mixtral 8x22B leads

Mistral Nemo: 28.5 (#296), Mixtral 8x22B: 36.9 (#262)

BenchmarkMistral NemoMixtral 8x22B
LMArena Text—1162
LMArena Creative Writing—1141
EQ-Bench Creative Writing881—
WildBench—71.1%
LMArena Multi-Turn—1130

Frequently asked questions

Is Mistral Nemo better than Mixtral 8x22B?

Mistral Nemo and Mixtral 8x22B score almost the same on the Noometry Index (26.4 vs 27.1), so choose on price, context window or the category you care about most.

Which is cheaper, Mistral Nemo or Mixtral 8x22B?

Mistral Nemo is cheaper. It lists at $0.15 per million input tokens and $0.15 per million output tokens; Mixtral 8x22B lists at $2 and $6.

Which has the bigger context window?

Mistral Nemo does, with 128K tokens against 64K.

How many benchmarks do Mistral Nemo and Mixtral 8x22B share?

4 benchmarks have published results for both models. Mistral Nemo has 10 scored results on Noometry and Mixtral 8x22B has 34.

Related comparisons

Go deeper