Nvidia is getting into world models — AI models that take inspiration from the mental models of the world that humans develop naturally.
At the Consumer Electronics Show in Las Vegas, the company announced that it is making openly available a family of world models that can predict and generate “physics-aware” videos. Nvidia’s calling this family Cosmos World Foundation Models, or Cosmos WFM for short.
The models, which can be fine-tuned for specific applications, are available from Nvidia’s API and NGC catalogs and the AI developer platform Hugging Face.
“Nvidia is making available the first wave of Cosmos WFMs for physics-based simulation and synthetic data generation,” the company wrote in a blog post provided to TechCrunch. “Researchers and developers, regardless of their company size, can freely use the Cosmos models under Nvidia’s permissive open model license that allows commercial usage.”
There are a number of models in the Cosmos WFM family, divided into three categories: Nano for low latency and real-time applications; Super for “highly performant baseline” models; and Ultra for maximum quality and fidelity output.
The models range in size from 4 billion to 14 billion parameters, with Nano being the smallest and Ultra being the largest. Parameters roughly correspond to a model’s problem-solving skills, and models with more parameters generally perform better than those with fewer parameters.
As a part of Cosmos WFM, Nvidia is also releasing an “upsampling model,” a video decoder optimized for augmented reality, and guardrail models to ensure responsible use, as well as fine-tuned models for applications like generating sensor data for autonomous vehicle development. These, as well as the other Cosmos WFM models, were trained on 9,000 trillion tokens from 20 million hours of real-world human interactions, environment, industrial, robotics, and driving data, Nvidia claimed. (In AI, “tokens” represent bits of raw data — in this case, video footage.)
Nvidia wouldn’t say where this training data came from, but at least one report — and lawsuit — alleges that the company trained on copyrighted YouTube videos without permission. We’ve reached out to Nvidia’s press team for comment and will update this piece if we hear back.
Nvidia claimed that Cosmos WFM models, given text or video frames, can generate “controllable, high-quality” synthetic data to bootstrap the training of models for robotics, driverless cars, and more.
“Nvidia Cosmos’ suite of open models means developers can customize the WFMs with data sets, such as video recordings of autonomous vehicle trips or robots navigating a warehouse, according to the needs of their target application,” Nvidia wrote in a press release. “Cosmos WFMs are purpose-built for physical AI research and development, and can generate physics-based videos from a combination of inputs, like text, image and video, as well as robot sensor or motion data.”
Nvidia said that companies including Waabi, Wayve, Fortellix, and Uber have already committed to piloting Cosmos WFMs for various use cases, from video search and curation to building AI models for self-driving vehicles.
Important to note is that Nvidia’s world models aren’t “open source” in the strictest sense. To abide by one widely accepted definition of “open source” AI, an AI model has to provide enough information about its design so that a person could “substantially” recreate it, and disclose any pertinent details about its training data, including the provenance and how the data can be obtained or licensed.
Nvidia hasn’t published Cosmos WFM training data details, nor has it made available all the tools needed to recreate the models from scratch. That’s probably why the tech giant is referring to the models as “open” as opposed to open source.
