I discovered at ICLR 2024 that a lot of what I take for granted about LLM evaluation is actually not that widely known...

So I made a blog!
- how do we do currently do LLM evaluation? ⚖️
- most importantly, what is it actually useful for? 🤔

huggingface.co/blog/clefourri…

Our recent paper 'Clinical Text Datasets for Medical Artificial Intelligence and Large Language Models — A Systematic Review' has been published in NEJM AI ! The lack of clinical text data is a long-term challenge/pain for #AIinMedicine and clinical LLM researchers. After

What are your favorite conferences/meetings/events for developers or researchers in healthcare working on or building with LLMs?

Beautiful aurora on the flight home from Symposium on AI for Learning Health Systems

AlphaFold 3 is out! As expected expands coverage to small molecules and nucleic acids. And replaces the structure module with a diffusion-based one. Unfortunately no code or model weights--just a web server for a limited set of ligands: blog.google/technology/ai/…

Full house for the LLM in Medicine panel at Symposium on AI for Learning Health Systems w/ Isaac Kohane Alan Karthikesalingam Bo Wang Katie Link Marlene Millen, MD

Even dialed in on Zoom…this workshop has a stellar lineup of sharp talks.

Day 1 of AI in Medicine Icahn School of Medicine at Mount Sinai Mount Sinai Health System has been a blast! Excited to delve into the past, present, and future of AI and its intersections with healthcare in the rest of the workshop this weekend. Glad to have the best co-organizers Jen Ren and Katie Link 🤩

So glad our workshop attendees (med students, residents, fellows) got to learn from Nishith Khandwala about what it takes to bring ML models from academia into clinical practice!

Teaching med students, residents, and fellows about adapting ML tasks to healthcare problems🔥

Really awesome talk by Akhil Vaid !

Medical and masters students often ask me how to quickly get up to speed on what AI 'is about' now (as opposed to a rich 70+ year history). So here are my very biased pointers: zaklab.org/blog/resources…

The big questions in medicine right now are less about the tech (though they ARE still about the tech), but about methodology, safety, integration (and HCI), and clinical trials.

Curious: what (proprietary/closed) LLMs are clinicians using on a day-to-day basis?

Please comment if something else!

Will do another poll for open LLMs 🙂

Today is my last day Hugging Face 🤗

It's been an incredible experience supporting open biomedical AI, working with seriously brilliant and kind colleagues, and collaborating with truly amazing scientists, engineers, and clinicians across the world. I've learned and grown so

enjoy earthquakes? also enjoy building stuff in healthcare? How about topicalness and contrived segues?

IT'S THE LAST DAY TO APPLY TO OUR HEALTHCARE AI HACKATHON IN SF!

We're getting people together to build generative AI stuff in healthcare!! In SF in June!

It'll be for 48

🧬 More than 170K researchers have downloaded our open source Nucleotide Transformer model in the past month, making it the most popular genomics model on Hugging Face 🤗 Try it here: huggingface.co/InstaDeepAI/nu…

While evaluation is challenging for many reasons (outcome and predictor definitions, time stamps, use case diffs), releasing open-source software to better standardize evals is 100% the right move by Epic.

Opens work up to both critique and improvement.

fiercehealthcare.com/ai-and-machine…