Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents

NVIDIA Nemotron 3 Nano Omni is a new omni-modal understanding model built for real-world document analysis, multiple image reasoning, automatic speech recognition, long audio-video understanding, agentic computer use, and general reasoning .

It extends the Nemotron multimodal line from a strong vision-language system to a broader text + image + video + audio model.

Nemotron 3 Nano Omni delivers best-in-class accuracy on complex document intelligence leaderboards such as MMlongbench-Doc , OCRBenchV2 , while also leading in video and audio leaderboards like WorldSense and DailyOmni . It achieves top accuracy on VoiceBench for audio understanding and ranks as the most cost‑efficient open video understanding model on MediaPerf.

Under the hood, it combines the Nemotron 3 hybrid Mamba-Transformer Mixture-of-Experts backbone with a C-RADIOv4-H vision encoder and Parakeet-TDT-0.6B-v2 audio encoder.

The architecture is designed to preserve fine visual detail, add native audio understanding, and scale to very long multimodal contexts for dense images, documents, videos, and mixed-modality reasoning.

The training recipe uses staged multimodal alignment and context extension , followed by preference optimization and multimodal reinforcement learning .

Nemotron 3 Nano Omni delivers up to 9x higher throughput and 2.9x the single-stream reasoning speed on multimodal use-cases, compared to alternatives.

Download the BF16 , FP8 and NVFP4 checkpoints at HuggingFace.

For more information about the model architecture, training recipe, data pipelines and benchmarks, read the full Nemotron 3 Nano Omni report .

Benchmark highlights Building on Nemotron Nano V2 VL, Nemotron 3 Nano Omni delivers substantial visual gains and adds entirely new audio and video+audio capabilities - while also leading another open-weights omni model, Qwen3-Omni, in many domains.

Efficiency highlights Compared to other open omni models with the same interactivity, Nemotron 3 Nano Omni delivers 7.4x higher system efficiency for multi-document use cases and 9.2x higher system efficiency for video use cases Figure 1. Total system throughput for multi-document and video use cases sustained by each model at a fixed per‑user interactivity threshold (tokens/sec/user)

What Nemotron 3 Nano Omni is designed for

At a high level, Nemotron 3 Nano Omni is aimed at five classes of workloads:

1. Real-world document analysis

This is not only about OCR. The model is positioned for long, messy, high-value documents where understanding depends on layout, tables, figures, formulas, section structure, and cross-page references. Think contracts, technical papers, reports, manuals, multi-page forms, or compliance packets. The model can handle 100+ page documents.

2. Automatic Speech Recognition

Nemotron 3 Nano Omni includes strong speech understanding capabilities that enable high-quality transcription across diverse audio conditions. It handles long-form audio with varying speakers, accents, and background noise. These capabilities can be integrated into broader workflows, allowing spoken content to be transcribed, analyzed, and combined with other modalities for tasks like summarization, question answering, and cross-modal reasoning.

3. Long audio-video understanding

Many enterprise and developer workflows depend on mixed audio and visual evidence: screen recordings with narration, training videos, meetings with slides, tutorials, product demos, customer support captures, and long-form video archives. Nemotron 3 Nano Omni is built to reason over those inputs jointly.

The Nemotron 3 Nano Omni model is specifically trained for agentic computer use, enabling it to assist with tasks in graphical user interface (GUI) environments. Its capabilities include interpreting screenshots, monitoring the state of the user interface, grounding its reasoning in on-screen visuals, and helping with action selection or workflow automation.

5. General multimodal reasoning

The model is designed for more than perception. It excels at reasoning-intensive tasks that require synthesizing information across long context windows, multiple modalities, and structured or semi-structured evidence. It can carry out multi-step reasoning, perform calculations, and connect signals from text, images, tables, and other inputs to arrive at coherent, well-supported answers.

Model architecture and key innovations

Nemotron 3 Nano Omni uses a unified encoder-projector-decoder design. The language backbone is Nemotron 3 Nano 30B-A3B, paired with the C-RADIOv4-H vision encoder and the Parakeet-TDT-0.6B-v2 audio encoder. The modality-specific encoders connect into the LLM backbone through lightweight projectors.

Figure 2. Model architecture of NVIDIA Nemotron 3 Nano Omni 30B-A3B

A hybrid Mamba-Transformer-MoE backbone for long multimodal context

The model backbone interleaves three key components: 23 Mamba selective state-space layers for efficient long-context processing; 23 MoE layers with 128 experts, top-6 routing , and a shared expert for conditional capacity; and 6 grouped-query attention layers to preserve strong global interaction and expressivity.

Nemotron 3 Nano Omni combines state-space models, attention, and MoE in a unified design that maintains strong reasoning performance while remaining practical for long, multimodal contexts.

Dynamic resolution for dense documents, charts, and screens

On the vision side, the Nemotron 3 Nano Omni replaces the tiling strategy used in the v2 model with dynamic resolution processing at native aspect ratio . Each image can be represented using a variable number of 16 x 16 patches, with a minimum of 1,024 to a maximum of 13,312 visual patches per image . For square images, this is equivalent to 512 x 512 and 1840 x 1840, respectively.

That flexibility is critical for handling high-resolution, complex visual inputs such as OCR-heavy documents, financial tables, slides, research figures, screenshots, and GUI layouts—especially when both fine details and overall structure need to be understood together.

Conv3D temporal compression for video

For video, Nemotron 3 Nano Omni uses a dedicated Conv3D tubelet embedding path. Instead of embedding each frame independently, every pair of consecutive frames is fused into a single "tubelet" before the ViT, halving the number of vision tokens the language model has to attend to. This allows us to either double the number of frames with the same token budget, or halve the number of tokens with the same number of frames

EVS — Efficient Video Sampling

EVS is an important feature, enabled during inference time, that drops redundant video tokens after the vision encoder. This reduces latency and improves throughput while maintaining accuracy. The first frame of the video is kept entirely, then for each subsequent frame, EVS keeps the “dynamic” tokens where the video is changing and drops the “static” ones where nothing has changed from the previous frame. We combine this with Conv3D to enable superior compression: Conv3D fuses tokens from pairs of frames into one, and then EVS prunes redundant static information.

Native audio input, not just text transcripts

The audio side is powered by Parakeet-TDT-0.6B-v2 , connected to the backbone through its own 2-layer MLP projector. Audio is sampled at 16 kHz , and the model is trained with inputs up to 1,200 seconds (20 minutes) , while the LLM max context length supports 5+ hours.

This represents a shift from traditional VLM pipelines by enabling native audio processing within a shared multimodal sequence, allowing audio, visual, and text tokens to be jointly modeled. This is crucial for scenarios like narrated screen recordings, video Q&A where speech alters visual meaning, long-form instructional or meeting content, and tasks requiring temporally grounded multimodal reasoning.

Lightweight modality projectors and unified token interleaving

Each encoder is connected to the LLM with a lightweight 2-layer MLP projector that maps encoder features into the shared embedding space. Once projected, vision, audio, and text tokens are interleaved and processed jointly .

This design keeps the overall system modular while still enabling genuine cross-modal reasoning inside the backbone itself.

Training data, infrastructure and systems story

The SFT stages are trained on NVIDIA H100 , scaling from 32 to 128 nodes depending on the stage. The stack uses Megatron-LM , Transformer Engine , and Megatron Energon , with tensor parallelism, expert parallelism, sequence parallelism, context parallelism for the long context stages, online sequence packing, and selective activation recomputation.

Post-SFT reinforcement learning uses NeMo-RL and NeMo Gym with a Megatron backend. The RL infrastructure used a Ray-based distributed setup across B200 and H100 clusters , plus multimodal deduplication, so repeated rollouts do not multiply image, video, and audio memory.

We open-source substantial parts of our training code.

Using RL to shape reliable multimodal behavior