Flan-2 is published in JMLR jmlr.org/papers/v25/23-…. I think it's a nice piece of history.

The work scaled instruction tuning with respect to model size and finetuning tasks, which both improved performance. Our MMLU was 75%, SOTA when the paper came out in Oct 2022.

Our…

In 2022, a model with 70%+ MMLU score, would cost 20 dollars per 1M tokens (instructGPT 3.5). Today it costs less than $1! It is perfectly reasonable to expect that in say five years, you will be able to use a model with 90%+ MMLU score for just a few cents per 1M tokens.

This new hallucinations eval by GDM friends is in the right direction in many ways:

1. Tackles the scenario of extremely long-form responses, which is a harder but more realistic setting
2. Extracts the number of relevant facts, then browses to verify each individual fact
3.…

Cheesy realization: studying history underscores how special this current moment in AI is. In past eras, the great powers of the world fought religious wars, sailed to unexplored lands, and built the first industrial cities. Now we will race to build artificial intelligence. So…

Had a bit of a fanboy moment today meeting Zero /dd, who has been super inspirational to me in prioritizing my health.

I asked him about the best way to balance career and spending time on health. His advice is that while many people give up sleep to work more, sleeping…

My mental model of Sora is that it is the “GPT-2 moment” for video generation.

GPT-2, which came out in 2018, could generate paragraphs of text that are coherent and grammatically correct. GPT-2 wasn’t able to write an entire essay without making mistakes like being inconsistent…

My typical day as a Member of Technical Staff at OpenAI:
[9:00am] Wake up
[9:30am] Commute to Mission SF via Waymo. Grab avocado toast from Tartine
[9:45 am] Recite OpenAI charter. Pray to optimization Gods. Learn the Bitter Lesson
[10:00am] Meetings (Google Meet). Discuss how to…

An incredible skill that I have witnessed, especially at OpenAI, is the ability to make “yolo runs” work.

The traditional advice in academic research is, “change one thing at a time.” This approach forces you to understand the effect of each component in your model, and…

A key insight from chain-of-thought is around the idea of information density. Language models can only do so much with a single forward pass, and so the amount of compute the language model can use must be scaled proportional to how hard a prompt is to solve.

What is…

One thing in AI research that I have finally recognized with clarity is the idea of “inertia bias”: continuing to do something when it’s not the best option.

The most basic instance of inertia bias is the feeling of “I already spent time implementing X, so let me continue trying…

There’s no adrenaline rush like launching a massive gpu training

For most companies, hiring more people is strictly better. However, this is often not true in AI research. AI research is often bottlenecked by compute, and when this is the case, hiring more researchers can be counter-productive.

I remember back at Google Brain, my manager once…

Browsing is great for information retrieval and massively reduces hallucinations, but I feel that it is easy for models that browse to lose some of the “magic” of large language models.

What I mean by magic is when language models give rich, organic responses reflecting the…

As an AI researcher there are many good reasons to write unit tests for your code, but perhaps the best motivator to write tests is the respect (and often surprise) from other people when they look at your code and see you actually wrote unit tests.

It’s inspiring to see co-founders & team leads at OpenAI (Greg being a prime example) writing code. The feeling invoked in me is almost like Medieval soldiers being inspired by the King fighting alongside them in battle (e.g., Richard the Lionheart, King Henry V, Charlemagne).

My 2024 goals:
- 1,000 hours writing code
- 100 thoughtful tweets
- 100 workouts
- 50 soccer practices
- 2,848 hours of sleep