Nvidia's new world model helps robots navigate the world

Nvidia unveiled Cosmos 3, an open AI world model designed to help robots, autonomous vehicles and other physical systems better understand and predict real-world environments.

Why it matters: Nvidia is continuing its move beyond chips into AI models and software, positioning itself to become a foundational platform for physical AI development.

Driving the news: Nvidia says it trained Cosmos 3 on 20 trillion tokens of multimodal data, including nearly a billion images, 400 million real and synthetic videos, ambient audio, text and action data from humans and robots.

• That action data is what makes Cosmos different from a regular video generator. It's meant to model how machines move, not just how scenes look, Ming-Yu Liu, VP of Nvidia's Cosmos Lab, told Axios. Autonomous actions are key.
• Developers can use Cosmos 3 to simulate actions in physical environments, then build task-specific models for robots and other machines on top of it.
• Cosmos 3 is designed to generate action data — such as robot joint angles, gripper positions and trajectories — that can help train machines to navigate and manipulate the physical world.

Between the lines: Cosmos is an open model, similar to its early Nemotron family, making it easier for hardware makers to customize Cosmos to their needs and ensure that future versions more closely align to the needs of the industry, Liu said.

• Nvidia is also establishing a coalition of companies supporting the effort. Initial partners include Agile Robots, Black Forest Labs and Runway.
• Nvidia says Cosmos can generate rare or dangerous scenarios — such as robot collisions or unusual road events — that are difficult, expensive or unsafe to capture repeatedly.

Zoom in: Nvidia is releasing two versions immediately: a "super" model for tasks requiring high physics accuracy, such as training robots and autonomous vehicles, and a "nano" model that can generate results in fractions of a second.

• An "edge" model that can run locally is coming soon, Nvidia said.

Zoom out: World models have become a key growth area for AI as companies increasingly want to take the smarts of chatbots and agents and allow them to perform real-world tasks.

• Among the hot startups in this area are Fei-Fei Li's World Labs and Yann LeCun's AMI Labs.
• "Ultimately what a world model wants to achieve is to help physical agents to become more generalizable," Liu said. "To become more generalizable, you need to understand the world so you understand how it works, so you can make a plan."

Bottom line: Nvidia's bet is that the next wave of AI won't just answer questions or generate images — it will need to predict, simulate and act in the physical world, and Nvidia wants its open models and infrastructure to be the place developers start.