NLPositionality: Characterizing Design Biases of Datasets and Models – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/03/01/nlp…

Beyond the Mud: Datasets, Benchmarks, and Methods for Computer Vision in Off-Road Racing – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2024/03/22/bey…

Very proud to be part of this Amazing MURI Team with Christos Papadimitriou Columbia University, Erik Demaine, Constantinos Daskalakis, Aleksander Madry Massachusetts Institute of Technology (MIT), Zico Kolter , Russ Salakhutdinov, Katia Sycara Complete Music Update, Kamalika Chaudhuri UC San Diego, and our own Soheil Feizi UMD Department of Computer Science.

1/ Given high-dim time series data (e.g., video, text), we'd like to know 'what happens in the future?' and 'what happened between A and B?' Turns out temporal contrastive learning provably answers these questions!

Paper+Code: arxiv.org/abs/2403.04082
Backstory and details below

You can do solid technical work or have long arguments about mumbo jumbo -- it is hard to do both. But one of them is much easier to do than the other.

Exciting personal update: I defended my PhD last week on Deep Learning on Graphs. Huge thanks to my committee Profs Tom Mitchell, Jure Leskovec, and my advisors Russ Salakhutdinov and Christos Faloutsos. I have joined Inflection AI to extend the benefits of intelligence to a wider audience!

Congratulations to my former CMU PhD student Zhilin Yang on his new LLM startup Moonshot AI, raising over $1B in VC funding.

Zhilin has done some fundamental work in NLP and large language models during his time at CMU including TransformerXL and XLNet. techcrunch.com/2024/02/21/moo…

I could not agree more with Philippe Beaudoin's response to Yann LeCun post. Quoting Philippe via Facebook:

'I find it fascinating that ML researchers talk about their intuition with such a high degree of confidence even after having been wrong so frequently in the past. I’m used to…

If only Sora were developed by Meta :)

Completely agree with Kevin: “It’s fine for creative tasks, but not (yet) autonomous agents” – the key word here is YET.

We are still far from having fully autonomous & reliable agents, but here are some early examples (albeit brittle) of agents navigating the web and…

lol

Check out a few demos and code of the multimodal GPT-4V + SoM agent from our VisualWebArena that you can run on any web task.

Code+Paper: jykoh.com/vwa

We've added a demo of the multimodal GPT-4V + SoM agent from VisualWebArena to our GitHub repo, and you can now run this agent on any web task you like!

Here is the agent helping us to find a good Thai restaurant in Pittsburgh (more examples in the 🧵):

Yann LeCun It doesn't count as a new invite, if the slide deck doesn't change.

<zing/>

most exciting paper *ever* from our Google AI lab at Princeton University: naman agarwal danielsuo Xinyi Chen
arxiv.org/abs/2312.06837
*** Convolutional filters predetermined by the theory, no learning needed! ***

I Tested a Next-Gen AI Assistant. It Will Blow You Away | WIRED wired.com/story/fast-for…

I will be giving a talk at the Winter School of Deep Learning, ISI Kolkata at 9:00am EST, 19:30 IST, 14:00 UTC, 4th Feb. The talk will be streamed at

youtube.com/live/_uUW8CTqm…

I will talk about embodied AI and will cover some of our recent work on AI Agents and VisualWebArena.

Tomorrow Prof. Salakhutdinov (Russ Salakhutdinov) will deliver a talk as a part of the plenary lecture series at Winter School of Deep Learning, ISI Kolkata. Join us on YT at 19:30 IST 4th Feb (14:00 UTC, 4th Feb) via- youtube.com/live/_uUW8CTqm…
#WSDL #LLM #LMM #multimodal #deeplearning

Thanks Will Knight and WIRED for highlighting our work on Web AI Agents and VisualWebArena:

wired.com/story/fast-for…

Paper + Code: jykoh.com/vwa

I think that in the near future we will have multimodal AI agents capable of planning, reasoning, and reliably executing…