Noometry

Model comparison

Mistral Small vs Tulu 3 (Tülu 3) 70B

Mistral Small and Tulu 3 (Tülu 3) 70B score almost the same on the Noometry Index (33.4 vs 33.0), so choose on price, context window or the category you care about most.

Last verified . 14 shared benchmarks.

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

Tulu 3 (Tülu 3) 70B Allen Institute for AI (Ai2)

33.0

Rank #251 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Mistral Small scores higher in 6 categories and Tulu 3 (Tülu 3) 70B in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.5 to 45.6.

Side by side

Mistral SmallTulu 3 (Tülu 3) 70B
ProviderMistral AIAllen Institute for AI (Ai2)
Noometry Index33.433.0
Released2024-02-262024-11-21
WeightsOpenOpen
Context window262K—
Max output256K—
Input $ / M tokens$0.15—
Output $ / M tokens$0.60—
Results tracked3914

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Tulu 3 (Tülu 3) 70B leads

Mistral Small: 34.0 (#247), Tulu 3 (Tülu 3) 70B: 36.0 (#214)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Coding13621235
SciCode26.5%—
BigCodeBench Instruct36.1%—
LiveBench Coding36.2%—
BigCodeBench Complete46.6%—
ALE-Bench497.62—

Agentic & Tool Use Not comparable

Mistral Small: 28.1 (#93), Tulu 3 (Tülu 3) 70B: —

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
Berkeley Function Calling Leaderboard37.1%—

Reasoning Tulu 3 (Tülu 3) 70B leads

Mistral Small: 19.8 (#250), Tulu 3 (Tülu 3) 70B: 23.9 (#169)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Hard Prompts13351220
Kagi LLM Benchmark37.8%—
CritPt0%—
LiveBench Reasoning44.8%—
DTBench70.9%—
LiveBench Data Analysis53.7%—
LMCA20.6%—
LiveBench44%—

Math Mistral Small leads

Mistral Small: 16.4 (#293), Tulu 3 (Tülu 3) 70B: 14.2 (#303)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
OTIS Mock AIME 2024-20255.8%4.4%
LMArena Math13411242
MATH Level 546.8%42.7%
LiveBench Math39.9%—

Knowledge Mistral Small leads

Mistral Small: 31.0 (#222), Tulu 3 (Tülu 3) 70B: 25.0 (#264)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
GPQA Diamond47.5%46.3%
Vectara Hallucination Rate5.1%—
LMArena Expert1291—
MMLU68.7%—

Multimodal Not comparable

Mistral Small: 33.5 (#96), Tulu 3 (Tülu 3) 70B: —

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Vision1142—

Multilingual Mistral Small leads

Mistral Small: 45.5 (#169), Tulu 3 (Tülu 3) 70B: 39.9 (#222)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Non-English13151236
LMArena Chinese13401249
LMArena Russian13241246
LMArena French1337—
LMArena German1340—
LMArena Japanese1275—
LMArena Korean1259—
LMArena Spanish1346—

Instruction Following Mistral Small leads

Mistral Small: 66.4 (#209), Tulu 3 (Tülu 3) 70B: 64.8 (#227)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Instruction Following13101233
LiveBench Instruction Following63.7%—

Long Context Mistral Small leads

Mistral Small: 40.4 (#156), Tulu 3 (Tülu 3) 70B: 37.1 (#222)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Longer Query13271224

Writing & Preference Mistral Small leads

Mistral Small: 52.5 (#171), Tulu 3 (Tülu 3) 70B: 45.6 (#223)

BenchmarkMistral SmallTulu 3 (Tülu 3) 70B
LMArena Text13381256
LMArena Creative Writing13051231
LMArena Multi-Turn13441252
LiveBench Language30.5%—

Frequently asked questions

Is Mistral Small better than Tulu 3 (Tülu 3) 70B?

Mistral Small and Tulu 3 (Tülu 3) 70B score almost the same on the Noometry Index (33.4 vs 33.0), so choose on price, context window or the category you care about most.

Is Mistral Small or Tulu 3 (Tülu 3) 70B better for coding?

Tulu 3 (Tülu 3) 70B scores higher on coding benchmarks: 36.0 versus 34.0 in the Noometry coding category.

How many benchmarks do Mistral Small and Tulu 3 (Tülu 3) 70B share?

14 benchmarks have published results for both models. Mistral Small has 39 scored results on Noometry and Tulu 3 (Tülu 3) 70B has 14.

Related comparisons

Go deeper