Noometry

Model comparison

Mistral Medium 3.5 vs Phi 3 Small 8k Instruct

Mistral Medium 3.5 is the stronger model overall, scoring 39.6 to 28.8 on the Noometry Index.

Last verified . 16 shared benchmarks.

Mistral Medium 3.5 Mistral AI

39.6

Rank #151 Confirmed

Phi 3 Small 8k Instruct Microsoft

28.8

Rank #310 Confirmed

Summary

  • They share 16 benchmarks with published results for both. Mistral Medium 3.5 scores higher in 8 categories and Phi 3 Small 8k Instruct in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Medium 3.5 leads 58.6 to 31.3.

Side by side

Mistral Medium 3.5Phi 3 Small 8k Instruct
ProviderMistral AIMicrosoft
Noometry Index39.628.8
Released—2024-04-23
WeightsOpenOpen
Context window262K—
Max output210K—
Input $ / M tokens$1.50—
Output $ / M tokens$7.50—
Results tracked2232

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Medium 3.5 leads

Mistral Medium 3.5: 35.7 (#208), Phi 3 Small 8k Instruct: 27.4 (#313)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Coding14611102
LMArena WebDev1263—
LiveBench Coding—20.3%

Reasoning Mistral Medium 3.5 leads

Mistral Medium 3.5: 16.6 (#290), Phi 3 Small 8k Instruct: 14.2 (#319)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Hard Prompts14361100
Kagi LLM Benchmark41.4%—
NYT Connections (extended)12.9%—
LiveBench Reasoning—15.9%
LiveBench Data Analysis—30.3%
Adversarial NLI—58.1%
BIG-Bench Hard—79.1%
Epoch Capabilities Index141.42—
HellaSwag—77%
LiveBench—24%
WinoGrande—81.5%

Math Mistral Medium 3.5 leads

Mistral Medium 3.5: 36.4 (#113), Phi 3 Small 8k Instruct: 25.4 (#247)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Math14321151
LiveBench Math—17.6%

Knowledge Mistral Medium 3.5 leads

Mistral Medium 3.5: 39.3 (#123), Phi 3 Small 8k Instruct: 28.5 (#237)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Expert14311067
ARC (AI2) Challenge—90.7%
MMLU—75.7%
OpenBookQA—88%
TriviaQA—58.1%

Multimodal Not comparable

Mistral Medium 3.5: 37.6 (#64), Phi 3 Small 8k Instruct: —

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Vision1222—

Multilingual Mistral Medium 3.5 leads

Mistral Medium 3.5: 52.1 (#93), Phi 3 Small 8k Instruct: 28.6 (#267)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Non-English14051058
LMArena Chinese14431061
LMArena French14481134
LMArena German14531080
LMArena Korean1387894
LMArena Russian13961111
LMArena Spanish14061111
LMArena Japanese—966

Instruction Following Mistral Medium 3.5 leads

Mistral Medium 3.5: 74.7 (#87), Phi 3 Small 8k Instruct: 52.2 (#287)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Instruction Following14141087
LiveBench Instruction Following—47.2%

Long Context Mistral Medium 3.5 leads

Mistral Medium 3.5: 43.2 (#101), Phi 3 Small 8k Instruct: 33.0 (#262)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Longer Query14141088

Writing & Preference Mistral Medium 3.5 leads

Mistral Medium 3.5: 58.6 (#113), Phi 3 Small 8k Instruct: 31.3 (#279)

BenchmarkMistral Medium 3.5Phi 3 Small 8k Instruct
LMArena Text14211110
LMArena Creative Writing13721083
LMArena Multi-Turn14231068
EQ-Bench 4993—
LiveBench Language—12.9%

Frequently asked questions

Is Mistral Medium 3.5 better than Phi 3 Small 8k Instruct?

Mistral Medium 3.5 is the stronger model overall, scoring 39.6 to 28.8 on the Noometry Index.

Is Mistral Medium 3.5 or Phi 3 Small 8k Instruct better for coding?

Mistral Medium 3.5 scores higher on coding benchmarks: 35.7 versus 27.4 in the Noometry coding category.

How many benchmarks do Mistral Medium 3.5 and Phi 3 Small 8k Instruct share?

16 benchmarks have published results for both models. Mistral Medium 3.5 has 22 scored results on Noometry and Phi 3 Small 8k Instruct has 32.

Related comparisons

Go deeper