Noometry

Model comparison

Claude 3 Sonnet vs Qwen2.5 7B Instruct

Claude 3 Sonnet and Qwen2.5 7B Instruct score almost the same on the Noometry Index (29.0 vs 29.0), so choose on price, context window or the category you care about most.

Last verified . 7 shared benchmarks.

Claude 3 Sonnet Anthropic

29.0

Rank #319 Confirmed

Qwen2.5 7B Instruct Alibaba (Qwen)

29.0

Rank #320 Confirmed

Summary

  • They share 7 benchmarks with published results for both. Claude 3 Sonnet scores higher in 2 categories and Qwen2.5 7B Instruct in 4 categories; 5 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Qwen2.5 7B Instruct leads 36.5 to 29.6.
  • The biggest single-benchmark swing is BigCodeBench Complete: 53.8% for Claude 3 Sonnet and 46.1% for Qwen2.5 7B Instruct.
  • Qwen2.5 7B Instruct has downloadable open weights; the other is API-only.

Side by side

Claude 3 SonnetQwen2.5 7B Instruct
ProviderAnthropicAlibaba (Qwen)
Noometry Index29.029.0
Released2024-02-292024-09
WeightsProprietaryOpen
Context window—131K
Max output—8K
Input $ / M tokens—$0.17
Output $ / M tokens—$0.70
Results tracked3015

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Qwen2.5 7B Instruct leads

Claude 3 Sonnet: 29.6 (#302), Qwen2.5 7B Instruct: 36.5 (#208)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
BigCodeBench Instruct42.7%37.6%
BigCodeBench Complete53.8%46.1%
WeirdML10.2%—
LMArena Coding1223—
HumanEval+64%—
MBPP+69.3%—

Agentic & Tool Use Not comparable

Claude 3 Sonnet: —, Qwen2.5 7B Instruct: 23.8 (#124)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
BALROG—7.8%

Reasoning Claude 3 Sonnet leads

Claude 3 Sonnet: 20.5 (#237), Qwen2.5 7B Instruct: 14.8 (#322)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
DTBench53.6%47.7%
Epoch Capabilities Index120.7118.51
Chess Puzzles—0%
LMArena Hard Prompts1197—
LMCA—6.4%
WinoGrande75.1%—

Math Qwen2.5 7B Instruct leads

Claude 3 Sonnet: 10.7 (#310), Qwen2.5 7B Instruct: 12.6 (#306)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
OTIS Mock AIME 2024-20252.5%2.5%
Omni-MATH—29.4%
LMArena Math1213—
MATH Level 518.2%—

Knowledge Claude 3 Sonnet leads

Claude 3 Sonnet: 21.1 (#276), Qwen2.5 7B Instruct: 17.0 (#286)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
GPQA Diamond40.6%35.5%
MMLU75.9%72.9%
MMLU-Pro—53.9%
GPQA (HELM)—34.1%
LMArena Expert1173—

Multimodal Not comparable

Claude 3 Sonnet: 25.2 (#125), Qwen2.5 7B Instruct: —

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
LMArena Vision984—

Multilingual Not comparable

Claude 3 Sonnet: 37.8 (#234), Qwen2.5 7B Instruct: —

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
LMArena Non-English1205—
LMArena Chinese1189—
LMArena French1229—
LMArena German1204—
LMArena Japanese1131—
LMArena Korean1128—
LMArena Russian1227—
LMArena Spanish1204—

Instruction Following Too close to call

Claude 3 Sonnet: 62.8 (#235), Qwen2.5 7B Instruct: 63.2 (#231)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
IFEval—74.1%
LMArena Instruction Following1199—

Long Context Not comparable

Claude 3 Sonnet: 36.7 (#228), Qwen2.5 7B Instruct: —

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
LMArena Longer Query1211—

Writing & Preference Qwen2.5 7B Instruct leads

Claude 3 Sonnet: 42.1 (#238), Qwen2.5 7B Instruct: 48.8 (#195)

BenchmarkClaude 3 SonnetQwen2.5 7B Instruct
LMArena Text1218—
LMArena Creative Writing1186—
WildBench—73.1%
LMArena Multi-Turn1227—

Frequently asked questions

Is Claude 3 Sonnet better than Qwen2.5 7B Instruct?

Claude 3 Sonnet and Qwen2.5 7B Instruct score almost the same on the Noometry Index (29.0 vs 29.0), so choose on price, context window or the category you care about most.

Is Claude 3 Sonnet or Qwen2.5 7B Instruct better for coding?

Qwen2.5 7B Instruct scores higher on coding benchmarks: 36.5 versus 29.6 in the Noometry coding category.

How many benchmarks do Claude 3 Sonnet and Qwen2.5 7B Instruct share?

7 benchmarks have published results for both models. Claude 3 Sonnet has 30 scored results on Noometry and Qwen2.5 7B Instruct has 15.

Related comparisons

Go deeper