Noometry

Model comparison

Mistral Nemo vs Mixtral 8x7B

Mistral Nemo and Mixtral 8x7B score almost the same on the Noometry Index (26.4 vs 27.1), so choose on price, context window or the category you care about most.

Last verified . 6 shared benchmarks.

Mistral Nemo Mistral AI

26.4

Rank #337 Confirmed

Mixtral 8x7B Mistral AI

27.1

Rank #334 Confirmed

Summary

  • They share 6 benchmarks with published results for both. Mistral Nemo scores higher in 3 categories and Mixtral 8x7B in 1 category; 4 gaps are clear of the uncertainty.
  • The widest gap is in math, where Mistral Nemo leads 25.5 to 18.8.
  • Mistral Nemo is cheaper at $0.15 / $0.15 per million input/output tokens, against $0.70 / $0.70 for Mixtral 8x7B.
  • Mistral Nemo accepts more context: 128K tokens versus 32K.

Side by side

Mistral NemoMixtral 8x7B
ProviderMistral AIMistral AI
Noometry Index26.427.1
Released2024-07-012023-12-11
WeightsOpenOpen
Context window128K32K
Max output128K32K
Input $ / M tokens$0.15$0.70
Output $ / M tokens$0.15$0.70
Results tracked1038

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Mistral Nemo: —, Mixtral 8x7B: 32.8 (#269)

BenchmarkMistral NemoMixtral 8x7B
LMArena Coding—1126
HumanEval+—39.6%
MBPP+—49.7%

Agentic & Tool Use Not comparable

Mistral Nemo: 23.5 (#125), Mixtral 8x7B: —

BenchmarkMistral NemoMixtral 8x7B
Berkeley Function Calling Leaderboard27.6%—
BALROG17.6%—

Reasoning Mistral Nemo leads

Mistral Nemo: 20.7 (#232), Mixtral 8x7B: 18.2 (#285)

BenchmarkMistral NemoMixtral 8x7B
DTBench48.6%49.6%
Epoch Capabilities Index118.68118.47
PIQA83.5%83.6%
LMArena Hard Prompts—1115
Adversarial NLI—55.2%
ForecastBench—56.3
HellaSwag—86.7%
WinoGrande—77.2%

Math Mistral Nemo leads

Mistral Nemo: 25.5 (#268), Mixtral 8x7B: 18.8 (#289)

BenchmarkMistral NemoMixtral 8x7B
MATH Level 510.8%10%
GSM8K84.2%74.4%
Omni-MATH—10.5%
LMArena Math—1147

Knowledge Mistral Nemo leads

Mistral Nemo: 12.3 (#298), Mixtral 8x7B: 11.0 (#301)

BenchmarkMistral NemoMixtral 8x7B
GPQA Diamond29.9%30.6%
MMLU-Pro—33.5%
GPQA (HELM)—29.6%
LMArena Expert—1088
ARC (AI2) Challenge—87.3%
BoolQ82.5%—
MMLU—70.6%
OpenBookQA—85.8%
TriviaQA—82.2%

Multilingual Not comparable

Mistral Nemo: —, Mixtral 8x7B: 29.6 (#266)

BenchmarkMistral NemoMixtral 8x7B
LMArena Non-English—1077
LMArena Chinese—1055
LMArena French—1166
LMArena German—1114
LMArena Japanese—931
LMArena Korean—968
LMArena Russian—1090
LMArena Spanish—1111

Instruction Following Not comparable

Mistral Nemo: —, Mixtral 8x7B: 51.0 (#297)

BenchmarkMistral NemoMixtral 8x7B
IFEval—57.5%
LMArena Instruction Following—1109

Long Context Not comparable

Mistral Nemo: —, Mixtral 8x7B: 33.4 (#260)

BenchmarkMistral NemoMixtral 8x7B
LMArena Longer Query—1103

Writing & Preference Mixtral 8x7B leads

Mistral Nemo: 28.5 (#296), Mixtral 8x7B: 34.2 (#270)

BenchmarkMistral NemoMixtral 8x7B
LMArena Text—1132
LMArena Creative Writing—1109
EQ-Bench Creative Writing881—
WildBench—67.3%
LMArena Multi-Turn—1115

Frequently asked questions

Is Mistral Nemo better than Mixtral 8x7B?

Mistral Nemo and Mixtral 8x7B score almost the same on the Noometry Index (26.4 vs 27.1), so choose on price, context window or the category you care about most.

Which is cheaper, Mistral Nemo or Mixtral 8x7B?

Mistral Nemo is cheaper. It lists at $0.15 per million input tokens and $0.15 per million output tokens; Mixtral 8x7B lists at $0.70 and $0.70.

Which has the bigger context window?

Mistral Nemo does, with 128K tokens against 32K.

How many benchmarks do Mistral Nemo and Mixtral 8x7B share?

6 benchmarks have published results for both models. Mistral Nemo has 10 scored results on Noometry and Mixtral 8x7B has 38.

Related comparisons

Go deeper