Po-Chen Ko Po-Chen Ko, an undergrad from our lab, who will apply for Fall 2025 PhD programs, will present our #ICLR2024 ICLR 2024 work. If you are interested in learning from videos or are hiring PhD students, come talk to him at #240 poster in the morning session on May 10.

Our work on UniSim (universal-simulator.github.io/unisim/) got an outstanding paper award!

Roughly 6 months before Sora, we showed how text-conditioned video models can be interactive simulators of the real world, enabling us to simulate long-horizon plans and train RL policies on robots.

Excited to share our #ICLR2024 spotlight paper “Compositional generative inverse design” (CinDM), a generative model that unifies simulation and inverse design for physical simulations, and can design parameters significantly more complex than in training: ai4s.lab.westlake.edu.cn/projects/cgid/

I'll be at ICLR 2024 next week!

Happy to chat about generative models, compositional models, robot learning and embodied agents.

I'll be presenting VLP (video-language-planning.github.io) showing how we can generate long-horizon video plans by combining VLMs + videos + search.

Excited to share work showing how we can construct compositional world models over the beliefs of an agent to enable embodied agents to collaborate!

By formulating generative modeling as an EBM, we can get substantial flexibility in both the generation procedure and the ability to generalize to new generative modeling problems.

We talk about the benefits of EBMs in:
youtube.com/watch?v=kFchGK…

Excited to share our work on constructing a 3D generative world model that can generate language, images and actions!

This enables us to imagine, plan and execute tasks across many environments.

Large video models will revolutionize physical decision-making!

In past work, we’ve shown how this enables us to construct universal policies (universal-policy.github.io), world simulators (universal-simulator.github.io) and long horizon planners (video-language-planning.github.io)!

MIT researchers propose that we should construct large generative models compositionally from systems of smaller models.

Robot learning has lots of heterogeneity from different sensor modalities (depth/tactile/rgb) and data modalities (simulation/real/human videos).

We illustrate how we can learn and combine information from all these sources by composing policies together!

Submit your ICML papers to our workshop on GenAI+Decision Making at ICLR 2024!

openreview.net/group?id=ICLR.…

First talk recording is available here:
youtu.be/0ACcDrWJs8I?si…

From Yilun Du

Pinterest presents RL Diffusion

Large-scale Reinforcement Learning for Diffusion Models

paper page: huggingface.co/papers/2401.12…

illustrate how this substantially improves pretrained Stable Diffusion (SD) models, generating samples that are preferred by humans 80.3% of the time…

Excited to share our work on compositional generative inverse design!

By composing different generative models together we design systems at test time much more complex than the examples seen in the training dataset:

openreview.net/forum?id=wmX0C…

A challenge in robotics research is the prevalence of misleading demos—those trained on test data, pre-scripted, or heavily tuned for a single environment. This can make genuinely innovative work seem less impressive, as people have seen more ‘impressive’ demos in the past.

How can we construct foundation models for robotic decision-making in the face of data scarcity? 🧵

MIT CSAIL, Anurag Ajay (anuragajay.github.io), Seungwook Han, Yilun Du, Pulkit Agrawal, and team address this with a compositional foundation model, which develops detailed,…

We're hosting a ICLR workshop on generative models for decision-making!

Generative models provide exciting new perspectives on solving decision-making settings such as planning, exploration, and inverse RL.

Submit your papers about this below:

openreview.net/group?id=ICLR.…

Come to foundation models for decision making workshop tommorow! We’ll have a great set of upcoming speakers!

We will be presenting 3D-LLM on Wednesday afternoon 5:00pm.

Happy to chat about embodied 3D foundation models at NeurIPS!