Noometry

Model comparison

ERNIE 5.1 vs Gemini 1.5 Flash (May 2024)

ERNIE 5.1 is the stronger model overall, scoring 43.8 to 33.2 on the Noometry Index.

Last verified . 17 shared benchmarks.

ERNIE 5.1 Baidu

43.8

Rank #83 Confirmed

Gemini 1.5 Flash (May 2024) Google

33.2

Rank #246 Confirmed

Summary

  • They share 17 benchmarks with published results for both. ERNIE 5.1 scores higher in 8 categories and Gemini 1.5 Flash (May 2024) in 0 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where ERNIE 5.1 leads 40.3 to 22.1.

Side by side

ERNIE 5.1Gemini 1.5 Flash (May 2024)
ProviderBaiduGoogle
Noometry Index43.833.2
Released—2024-05-14
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1942

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding ERNIE 5.1 leads

ERNIE 5.1: 44.0 (#76), Gemini 1.5 Flash (May 2024): 34.4 (#236)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Coding14881261
WeirdML—24.9%
BigCodeBench Instruct—43.5%
BigCodeBench Complete—55.1%
HumanEval+—75.6%
MBPP+—67.5%

Agentic & Tool Use Not comparable

ERNIE 5.1: —, Gemini 1.5 Flash (May 2024): 26.6 (#102)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
BALROG—14.6%
LMArena Search1227—

Reasoning Too close to call

ERNIE 5.1: 21.9 (#211), Gemini 1.5 Flash (May 2024): 21.7 (#215)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Hard Prompts14811257
NYT Connections (extended)23.4%—
DTBench—53.8%
Epoch Capabilities Index—129.36
ForecastBench—53.9
PIQA—87.5%

Math ERNIE 5.1 leads

ERNIE 5.1: 40.3 (#92), Gemini 1.5 Flash (May 2024): 22.1 (#281)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Math14811269
OTIS Mock AIME 2024-2025—16.3%
Omni-MATH—30.4%
MATH Level 5—61.9%
FrontierMath (Feb 2025 set)—0%
GSM8K—82.4%

Knowledge ERNIE 5.1 leads

ERNIE 5.1: 41.9 (#102), Gemini 1.5 Flash (May 2024): 26.2 (#260)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Expert14921233
GPQA Diamond—47.3%
MMLU-Pro—67.8%
GPQA (HELM)—43.7%
BoolQ—85.8%
MMLU—77.9%

Multimodal Not comparable

ERNIE 5.1: —, Gemini 1.5 Flash (May 2024): 36.0 (#81)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Vision—1141
Video-MME—70.3%
GeoBench—76%

Multilingual ERNIE 5.1 leads

ERNIE 5.1: 55.5 (#29), Gemini 1.5 Flash (May 2024): 42.9 (#189)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Non-English14541278
LMArena Chinese15081295
LMArena French14881258
LMArena German14701262
LMArena Japanese14221252
LMArena Korean14271221
LMArena Russian14591288
LMArena Spanish14731243

Instruction Following ERNIE 5.1 leads

ERNIE 5.1: 76.7 (#37), Gemini 1.5 Flash (May 2024): 66.8 (#205)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Instruction Following14601258
IFEval—83.1%

Long Context ERNIE 5.1 leads

ERNIE 5.1: 44.7 (#59), Gemini 1.5 Flash (May 2024): 39.0 (#187)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Longer Query14621284

Writing & Preference ERNIE 5.1 leads

ERNIE 5.1: 65.1 (#52), Gemini 1.5 Flash (May 2024): 48.7 (#196)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Text14681287
LMArena Creative Writing14411285
LMArena Multi-Turn14711253
WildBench—79.2%

Frequently asked questions

Is ERNIE 5.1 better than Gemini 1.5 Flash (May 2024)?

ERNIE 5.1 is the stronger model overall, scoring 43.8 to 33.2 on the Noometry Index.

Is ERNIE 5.1 or Gemini 1.5 Flash (May 2024) better for coding?

ERNIE 5.1 scores higher on coding benchmarks: 44.0 versus 34.4 in the Noometry coding category.

How many benchmarks do ERNIE 5.1 and Gemini 1.5 Flash (May 2024) share?

17 benchmarks have published results for both models. ERNIE 5.1 has 19 scored results on Noometry and Gemini 1.5 Flash (May 2024) has 42.

Related comparisons

Go deeper