While studying memorization in LLMs, we struggled with a basic question: can knowledge really be localized to specific parameters? So, we designed two independent benchmarks to answer this. Amazingly, they agree: pruning-based localization is best, works well, but isn’t perfect.

Localization in LLMs is often mentioned. But do localization methods actually localize correctly? In our #NAACL2024 paper, we (w/ Jesse Thomason, Robin Jia) develop two benchmarking ways to directly evaluate how well 5 existing methods can localize memorized data in LLMs.

Do multimodal foundation models treat every modality equally?

Hint: Humans have picture superiority. How about machines?

Introducing IsoBench, a benchmark for multimodal models with isomorphic inputs.

🔗 IsoBench.github.io

Join us for the USC Symposium on Frontiers of Generative AI Models in Science and Society!

Featuring special guests Alessandro Vespignani Nitesh Chawla Yizhou Sun Jian Ma & Robin Jia Yue Wang Ruishan Liu USC Viterbi School

📅 Mar 25
📍MCB 101
🔗RSVP (limited space) tinyurl.com/4zpmfysa

Personally my favorite experiment from this data watermark work. BLOOM-176B memorizes SHA hashes from its training data; we can use similar random character sequences to watermark any document collection!

Determining whether an LLM has trained on your data isn’t a classification problem, it’s a statistical testing problem. Really proud of this work by Johnny Tian-Zheng Wei and Ryan Yixiang Wang on using random watermarks to rigorously detect data usage!

🔊 Please consider submitting to our Secure and Trustworthy Large Language Model Workshop at #ICLR2024 ! Submission ddl is Feb 12.

ACL announcement:
'The ACL Executive Committee has voted to significantly change ACL's approach to protecting anonymous peer review. The change is effective immediately. (1/4) #NLPRoc

I’m at #NeurIPS23 and on the job market🎷🧳!! Come and talk about anything LLM reasoning, evaluating communicating agents, human-AI collaboration for new discoveries, coffee and jazz in NOLA☕️

hey EMNLP 2024 #EMNLP2023 i lost my bucket hat 😢 it’s white w yellow bananas 🍌 on it like in my profile, pls DM me if found 🙏🏻

I’ll be presenting SCENE at the poster session this *Saturday* at 11am! Come chat about unanswerable questions, minimal pairs, extrapolation, and the value of synthetic data :)

Sadly I won't be at #EMNLP2023 in person. However, my amazing advisor Robin Jia will be presenting this work at the poster session!

We're hiring! USC Thomas Lord Department of Computer Science is growing fast, with multiple openings for tenure-track and tenured positions.

🔍Security/privacy, AI, machine learning, data science, HCI, but exceptional candidates in all areas considered.

📅Deadline: Jan 5
🔗Details: tinyurl.com/5e99bmb8

🚨Now hiring full-time teaching faculty!🚨

Apply now to join our fast-growing, collaborative department, with a brand new building opening this spring.

Find out more about our open faculty positions
(teaching, tenured, tenure track): cs.usc.edu/about/open-fac…

Please share!

We are at #EMNLP2023 this week! 👋

Explore the latest USC Thomas Lord Department of Computer Science USC ISI research spanning social bias in name translation, AI tools for journalists, ambiguity in LMs, and much more ⬇️

viterbischool.usc.edu/news/2023/12/u…

USC Viterbi School EMNLP 2024 USC Research #NLP

I just arrived in Singapore for #EMNLP2023 ! 😜

I'll be presenting our work on modeling and predicting LLM capabilities at GenBench workshop poster session (10am tomorrow). Also presenting it virtually in gather.town at 4pm on Dec 8. Check out our thread below 👇

It's a great honor to receive this award! Had so much fun gathering together at SoCalNLP today! #SoCalNLP2023 🎉

What can we learn from thousands of LLM experiment records? Can we predict emergent abilities? Does this give any clue on making LLM benchmarking more efficient and principled? Check out our paper on analyzing BIG-bench and designing “small-bench.”

arxiv.org/abs/2305.14947 🧵⬇️

ICL reduces the need for labeled *training* data, but what about test data? Fantastic work led by USC undergrad Harvey Yiyun Fu (applying to PhD programs this year!) shows we can estimate ICL accuracy well (on par with 40 labeled examples) using *unlabeled* data + model uncertainties

How do Transformers really do in-context linear regression? 1 TF layer = 3 steps of a second-order method! Can’t be GD, which converges exponentially more slowly. Meanwhile, LSTMs are more like online GD; they don’t learn second-order optimization (likely due to limited memory).