Cosmos-Transfer1-7B-Sample-AV-Single2MultiView: A Cosmos Transfer Sample Focusing on Autonomous Driving for View Extension

Cosmos | Code | Paper | Paper Website

Model Overview

Description:

Cosmos Transfer Sample Autonomous Driving Models: Cosmos Sample Autonomous Driving Models are a family of high-performance, post-trained Cosmos foundation models designed specifically for autonomous driving scenarios.

Cosmos-Transfer1-7B-Sample-AV-Single2MultiView Models: These models are fine-tuned versions of the Cosmos World foundation models, enabling them to translate front-view driving videos into multi-view consistent, high-quality driving videos. They serve as versatile building blocks for a wide range of applications and research related to autonomous driving. Ready for commercial use, the models are available under the NVIDIA Open Model License Agreement.

Model Developer: NVIDIA

Model Versions

The Cosmos Diffusion-based Control model family includes the following models:

• [Cosmos-Transfer1-7B-Sample-AV-Single2MultiView-Text2World [HDMap]]
  • Given a front-view video, text description and multi-view HDMAP video as input, generate 57 frames of video at resolution of 576x1024.
• [Cosmos-Transfer1-7B-Sample-AV-Single2MultiView-Text2World [LiDAR]]
  • Given a front-view video, text description and multi-view LIDAR video as input, generate 57 frames of video at resolution of 576x1024.
• [Cosmos-Transfer1-7B-Sample-AV-Single2MultiView-Video2World [HDMap]]
  • Given a front-view video, multiview initial frames video, text description and multi-view HDMAP video as input, generate 57 frames of video at resolution of 576x1024.
• [Cosmos-Transfer1-7B-Sample-AV-Single2MultiView-Video2World [LiDAR]]
  • Given a front-view video, multiview initial frames video, text description and multi-view LIDAR video as input, generate 57 frames of video at resolution of 576x1024.

License:

This model is released under the NVIDIA Open Model License. For a custom license, please contact cosmos-license@nvidia.com.

Under the NVIDIA Open Model License, NVIDIA confirms:

• Models are commercially usable.
• You are free to create and distribute Derivative Models.
• NVIDIA does not claim ownership to any outputs generated using the Models or Derivative Models.

Important Note: If You bypass, disable, reduce the efficacy of, or circumvent any technical limitation, safety guardrail or associated safety guardrail hyperparameter, encryption, security, digital rights management, or authentication mechanism (collectively “Guardrail”) contained in the Model without a substantially similar Guardrail appropriate for your use case, your rights under this Agreement NVIDIA Open Model License Agreement will automatically terminate.

Deployment Geography:

Global

Use Case:

Research related to autonomous driving; enabling users to generate high-quality driving videos from text, image, or video inputs.

Release Date:

Huggingface [05/15/2025]

Model Architecture:

Cosmos-Transfer1-7B-Sample-AV-Single2Multiview is a diffusion transformer model designed for video denoising in the latent space. The network is composed of interleaved self-attention, cross-attention and feedforward layers as its building blocks. The cross-attention layers allow the model to condition on input text throughout the denoising process. Before each layer, adaptive layer normalization is applied to embed the time information for denoising. When image or video is provided as input, their latent frames are concatenated with the generated frames along the temporal dimension.

Input/Output Specifications

• Input

  • Input Type(s): Text+Video
  • Input Format(s):
    • Text: String
    • Video: mp4
  • Input Parameters:
    • Text: One-dimensional (1D)
    • Front-view video: Three-dimensional (3D)
    • HDMAP/Lidar multi-view control video: Three-dimensional (3D)
    • Initial frame multi-view video: Three-dimensional (3D)
  • Other Properties Related to Input:
    • The input string should contain fewer than 300 words and should provide descriptive content for world generation, such as a scene description, key objects or characters, background, and any specific actions or motions to be depicted within the 2-second duration.
    • The input video should be of 1024x576 resolution.

• Output

  • Output Type(s): Video
  • Output Format(s): mp4
  • Output Parameters: Three-dimensional (3D)
  • Other Properties Related to Output: By default, the generated video is the same length and frames-per-second (fps) as the input with a resolution of 1024x576 pixels. The video content transforms the input video into a photorealistic multi-view scene, following the input text description.

Software Integration

Runtime Engine(s):

• Cosmos-Transfer1

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper

Operating System(s):

• Linux (We have not tested on other operating systems.)

Usage

• See Cosmos-Transfer1 for inference details.

Note: We have only tested doing inference with BF16 precision.

Evaluation

Please see our technical paper for detailed evaluations.

Data Collection Method:

• AV: Automatic/Sensors

Labeling Method:

• AV: Hybrid: Human, Automated

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.

For more detailed information on ethical considerations for this model, please see the subcards of Explainability, Bias, Safety & Security, and Privacy below. Please report security vulnerabilities or NVIDIA AI Concerns here.

Plus Plus (++) Promise

We value you, the datasets, the diversity they represent, and what we have been entrusted with. This model and its associated data have been:

• Verified to comply with current applicable disclosure laws, regulations, and industry standards.
• Verified to comply with applicable privacy labeling requirements.
• Annotated to describe the collector/source (NVIDIA or a third-party).
• Characterized for technical limitations.
• Reviewed to ensure proper disclosure is accessible to, maintained for, and in compliance with NVIDIA data subjects and their requests.
• Reviewed before release.
• Tagged for known restrictions and potential safety implications.

Bias

Explainability

Privacy

Safety