amazing work!

We hope you all have fun with it! Finally, its public version 'gpt-4o' is now up to the Arena. Come chat & vote at chat.lmsys.org!

Learn more in GPT-4o release blog post
openai.com/index/hello-gp…

It also demonstrates strong performance via community vibe-check. Check out cool gpt2-chatbot demos here.
x.com/minchoi/status…

Significantly higher win-rate against all other models.
e.g., ~80% win-rate vs GPT-4 (June) in non-tie battles.

In more challenging Coding Arena, we see even bigger gap (~100 Elo)!

Confidence intervals chart — huge gap against previous top-5 models.

Breaking news — gpt2-chatbots result is now out!

gpt2-chatbots have just surged to the top, surpassing all the models by a significant gap (~50 Elo). It has become the strongest model ever in the Arena!

With improvement across all boards, especially reasoning & coding…

GPT-4o is our new state-of-the-art frontier model. We’ve been testing a version on the LMSys arena as im-also-a-good-gpt2-chatbot 🙂. Here’s how it’s been doing.

4. Qualitatively, we also find Llama 3’s outputs are friendlier and more conversational than other models. These traits appear more often in battles that Llama 3 wins.

Llama 3 also really loves exclamations!

3. Deduplication or outliers do not significantly affect the win rate.

We also sanity-check votes and prompts to avoid certain users being over-represented. Results show that there's no change on Llama 3's win rate before/after.

(Cont'd) We show Llama 3-70b-Instruct's win rate conditioned on hierarchical criteria subsets. Some criteria separate the model's strengths and weaknesses.

2. As prompts get challenging*, the gap between Llama 3 against top-tier models becomes larger.

* We define challenging using several criteria like complexity, problem-solving, domain knowledge, and more.

Exciting new blog -- What’s up with Llama-3?

Since Llama 3’s release, it has quickly jumped to top of the leaderboard. We dive into our data and answer below questions:

- What are users asking? When do users prefer Llama 3?
- How challenging are the prompts?
- Are certain users…