rewatched Her last weekend and it felt a lot like rewatching Contagion in Feb 2020

GPT-4o is really good

I wish every grad student in AI would read The Bitter Lesson incompleteideas.net/IncIdeas/Bitte…

I’m at ICLR! Looking forward to catching up with friends, meeting new folks, and trying some Viennese schnitzel

Well said. There is a big opportunity for a neutral third party like Scale AI to step in as the 'Moody's of LLMs' and provide rigorous and comprehensive evals of all models.

Data contamination is a huge problem for LLM evals right now. At Scale, we created a new test set for GSM8k *from scratch* to measure overfitting and found evidence that some models (most notably Mistral and Phi) do substantially worse on this new test set compared to GSM8k.

Llama 3 is out in 8B and 70B sizes! (400B still training) Congrats to the AI at Meta team! ai.meta.com/blog/meta-llam…

Too many startups focused on what GPT-4 isn't, not enough startups focused on what future models could be

Someone on the admissions committee for a top CS PhD program told me they no longer filter based on paper count because too many of the applicants already have multiple publications. Instead, they now filter by citation count. Not sure if he was joking but I believed it.

Eval numbers for the new GPT-4 Turbo are out

🔥Exciting news -- GPT-4-Turbo has just reclaimed the No. 1 spot on the Arena leaderboard again! Woah!

We collect over 8K user votes from diverse domains and observe its strong coding & reasoning capability over others. Hats off to OpenAI for this incredible launch!

To offer…

The new GPT-4-Turbo improves an impressive 4.5 points on LiveCodeBench (comprising competition-style programming problems).

These problems are quite challenging for current LLMs and this improvement highlights a considerable improvement in reasoning!!

x.com/polynoamial/st…

I tried to get a friend to read the scaling laws paper but they said it was too long so I sent them this gif instead

Eliezer Yudkowsky ⏹️ while computers may excel at soft skills like creativity and emotional understanding, they will never match human ability at dispassionate, mechanical reasoning