Qwen3-Omni Technical Report | Summary

This article explains the key points of Qwen3-Omni Technical Report.

• 2025-09-22 (arXiv)
• Xu, Jin, Guo, Zhifang, Hu, Hangrui, Chu, Yunfei, Wang, Xiong, He, Jinzheng, Wang, Yuxuan, Shi, Xian, He, Ting, Zhu, Xinfa, et al.
• Paper
• Github

Read this article in Korean

Summary

• Qwen3-Omni addresses how to combine text, image, audio, and video processing in one model without weakening capabilities developed by models specialized for individual input types. In a controlled comparison of equally sized Qwen base models with matched training schedules, shared modality datasets, and training compute, it broadly preserves text and vision performance while adding audio and audiovisual capabilities, although some individual benchmark scores decline.
• The model separates a Thinker, which interprets inputs and generates text, from a Talker, which generates speech; both use Mixture-of-Experts (MoE), architectures that selectively activate expert components. Audio Transformer (AuT), the audio encoder, and speech generation operate at 12.5 Hz, while multiple codebooks, sets of discrete codes representing each audio frame, support waveform streaming through a lightweight causal convolutional network that uses only current and preceding context. The reported theoretical cold-start first-packet latency, the time to produce the first speech packet without prior context, is 234 ms for audio and 547 ms for video at one concurrent request.
• The report claims best open-source results on 32 of 36 audio and audiovisual benchmarks and best overall results on 22, with support for 119 text languages, 19 speech-input languages, and 10 speech-output languages. The Instruct variant, trained to respond to instructions, generally performs better on transcription and music perception, while the Thinking variant, which explicitly reasons over inputs, improves several reasoning evaluations. Weak long-video results, the absence of a comprehensive comparison across model scales, and primarily text-conditioned speech evaluation limit these conclusions.

1 Introduction

The introduction identifies modality trade-offs as an obstacle for integrated systems built around large language models (LLMs), models that process and generate language. It proposes mixing single-modality and cross-modality data early in pretraining to preserve specialized capabilities while developing audiovisual reasoning and interaction. Relative to Qwen2.5-Omni, the principal changes are MoE Thinker and Talker modules, a new audio encoder, richer speech representations, frame-level waveform generation, and lower audio token rates.

2 Architecture

The architecture combines audio and vision encoders with the Thinker–Talker interface and a streaming speech decoder. Representing acoustic detail in multiple discrete codebooks allows the final waveform renderer to use a lightweight convolutional network.

2.1 Overview

Thinker generates textual responses, while Talker receives audio and visual features from Thinker together with discrete text tokens and the conversation history. Talker no longer consumes Thinker’s high-level text representations, allowing separate system prompts for textual response style and speech style and external processing of generated text before synthesis. Talker predicts the first codebook for each frame, a multi-token prediction (MTP) module predicts the remaining codebooks, and Code2Wav, the waveform renderer, converts them into audio samples.

The examples organize the intended capabilities around reasoning, multilingual interaction, response speed, and long audio. They illustrate the model’s intended uses; the benchmark tables provide quantitative evidence.

The diagram connects perceptual encoders to Thinker and then to Talker, showing how speech generation uses multimodal features and text. The codebook prediction and streaming decoder path explains why waveform generation can begin after one frame.

2.2 Audio Transformer (AuT)

AuT is an attention-based encoder–decoder trained from scratch on 20 million hours of supervised audio: 80% Chinese and English pseudo-labeled automatic speech recognition (ASR), transcription of speech into text using automatically generated training labels, 10% ASR in other languages, and 10% audio understanding. Convolutional downsampling reduces the input feature rate eightfold to 12.5 Hz, and dynamic attention windows span 1–8 seconds to balance streaming computation with offline understanding. The prose describes an approximately 0.6B-parameter encoder, while Table 1 lists 650M parameters.

Three downsampling convolutional blocks precede 32 encoder attention layers, producing 12.5 Hz audio representations. The eight-layer decoder belongs to AuT training, while Qwen3-Omni uses the encoder for audio input.

2.3 Perceivation

Text uses byte-level byte-pair encoding, a tokenizer that merges frequently occurring byte sequences, with 151,643 regular tokens. Audio is resampled to 16 kHz and transformed into a 128-channel mel-spectrogram, a time–frequency representation, using a 25 ms window and 10 ms hop; each encoded frame represents approximately 80 ms. The vision encoder comes from Qwen3-VL, is initialized from SigLIP2-So400m, a vision-language encoder, and has approximately 543 million parameters in the prose; video frames are sampled at a dynamic rate.

• Time-aligned Multimodal Rotary Position Embedding (TM-RoPE), a position encoding that combines absolute time with spatial coordinates, assigns interleaved allocations of 24, 20, and 20 rotary angles to time, height, and width.
• Audio positions advance every 80 ms, and video positions follow actual timestamps at the same temporal resolution. This directly aligns audiovisual features instead of grouping them into fixed 2-second chunks.
• Text shares position identifiers across the three dimensions; images use a constant temporal identifier with spatial row and column positions. Each subsequent modality starts at one plus the preceding modality’s maximum position identifier.

2.4 Speech Generation

Talker conditions speech on historical text tokens, multimodal representations, and the current streamed response so that acoustic style can follow the conversation. It operates on residual vector quantization (RVQ) tokens, discrete codes representing successive residual acoustic detail: the backbone predicts the zeroth codebook, and MTP predicts the remaining codebooks. A causal ConvNet, the convolutional waveform renderer introduced above, reconstructs audio without waiting for a larger block of future context.

2.5 Designs for Streaming and Concurrency

Streaming combines asynchronous chunked prefilling, processing incoming context in overlapping Thinker and Talker stages before output generation, with left-context-only speech decoding and lightweight batched MTP and waveform modules. Using vLLM, the model-serving framework, with torch.compile and CUDA Graph acceleration for the speech modules, Table 2 reports theoretical audio/video first-packet latencies of 234/547 ms, 728/1517 ms, and 1172/2284 ms at 1, 4, and 6 concurrent requests. Startup latency increases with concurrency, while the reported generation rates remain faster than real time.

• Table 1 lists a 30B-A3B Thinker, a 3B-A0.3B Talker, an 80M MTP module, and a 200M waveform decoder; the model identifiers distinguish total and active parameter counts.
• The generation real-time factor (RTF), processing time divided by produced audio duration, is 0.47, 0.56, and 0.66 at the three concurrency levels, all below 1.
• RTF sums Thinker and Talker per-token generation time with MTP and decoder processing time, then divides by the 80 ms audio duration represented by one frame. The report does not identify the inference hardware used for this table.

The component breakdown separates the large Thinker from the smaller Talker and lightweight speech modules. It marks the vision encoder as non-streaming while listing streaming support for the other components and audio/video first-packet latency of 234/547 ms.

Startup latency rises from 234/547 ms at one concurrent request to 1172/2284 ms at six, while real-time factor remains below 1. The table distinguishes response startup costs from sustained speech-generation speed.

3 Pretraining

Pretraining mixes text, image–text, video–text, audio–text, video–audio, and video–audio–text data with varied natural-language prompts. Encoder Alignment Stage (S1) holds the language model fixed and trains adapters, modules connecting encoder outputs to the language model, before training the encoders; General Stage (S2) unfreezes all parameters. Long Context Stage (S3) increases maximum sequence length from 8,192 to 32,768 and raises the proportion of long audio and video.

• S2 is described as approximately 2 trillion tokens, with reported modality totals of 0.57 trillion text, 0.77 trillion audio, 0.82 trillion image, 0.05 trillion video, and 0.05 trillion video–audio tokens. These listed values sum to 2.26 trillion.
• Speech input supports ar, de, en, es, fr, id, it, ja, ko, ms, nl, pt, ru, th, tr, ur, vi, yue, zh; speech output supports de, en, es, fr, it, ja, ko, pt, ru, zh.
• The authors motivate adapter-first alignment by the risk that jointly training an encoder and adapter against a frozen language model could weaken the encoder’s perception capabilities.

Language coverage differs substantially between written interaction, speech understanding, and speech generation. Korean appears in both speech lists, but only 10 languages support speech output.

4 Post-training

Post-training uses separate pipelines for Thinker and Talker, with an additional fine-tuned model for audio captioning, generating written descriptions of sounds. Dialogue data use ChatML, the conversation formatting scheme cited by the report, across text-only and multimodal interactions.

4.1 Thinker

Thinker undergoes supervised fine-tuning (SFT), training on target responses, strong-to-weak distillation, transferring behavior from larger teacher models, and Group Sequence Policy Optimization (GSPO), the reinforcement-learning method used to optimize multimodal responses with rewards. Distillation first uses teacher-generated responses, then aligns the student’s output distributions with Qwen3-32B or Qwen3-235B-A22B on student-generated sequences by minimizing Kullback–Leibler divergence, a measure of distribution mismatch.

• Verifiable tasks use rule-based rewards for correctness and instruction compliance.
• Tasks without predefined objective metrics use language models as automated judges: Qwen3 for general tasks and Qwen2.5-VL for visually grounded tasks, with reference answers supplied where applicable.

4.2 Talker

Talker training has four stages: learning speech from hundreds of millions of contextual speech examples, continual pretraining on higher-quality data with long-context training, multilingual preference optimization, and speaker-specific fine-tuning. Direct Preference Optimization (DPO), training from preferred and dispreferred speech pairs, targets multilingual generalization and stability. The final stage adapts voices and refines naturalness, expressiveness, and controllability.

4.3 Captioner

Qwen3-Omni-30B-A3B-Captioner is obtained by fine-tuning Qwen3-Omni-30B-A3B on a large-scale dataset of detailed audio descriptions. The authors describe its captions as detailed and low in hallucination, unsupported generated content, but provide three qualitative examples rather than a quantitative hallucination evaluation.

5 Evaluation

Evaluation covers Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Thinking, and the in-house Qwen3-Omni-Flash-Instruct and Qwen3-Omni-Flash-Thinking variants. Results are organized into X→Text, textual responses to inputs of different modalities, and X→Speech, speech generation. Flash variants incorporate additional functionality, including dialect support, so their scores must be distinguished from those of the released 30B-A3B models.

5.1.1 Performance of Text→Text

Evaluation of X→Text spans general knowledge, reasoning, coding, instruction following, function calling, multilingual tasks, audio perception, visual understanding, and audiovisual reasoning. On text tasks, Instruct exceeds GPT-4o-0327 on GPQA, graduate-level questions, AIME25, competition mathematics, ZebraLogic, logical reasoning, WritingBench, writing quality, and PolyMATH, multilingual mathematics, but results against the same-size text-only Qwen model are mixed. Thinking improves several scores over Omni-Instruct while remaining below the same-size text-only Thinking model on most shared evaluations.

• On AIME25, Omni-Instruct scores 65.0 versus 61.3 for Qwen3-30B-A3B-Instruct-2507; on ZebraLogic, the comparison reverses to 76.0 versus 90.0.
• Omni-Thinking scores 73.7 on AIME25 versus 85.0 for Qwen3-30B-A3B-Thinking-2507, and 63.2 on BFCL-v3, a function-calling evaluation, versus 72.4.
• WritingBench favors Omni-Thinking at 85.5 versus 85.0 for the same-size text-only Thinking model. The tables do not support equal performance on every task.

Omni-Instruct exceeds the same-size text-only model on AIME25 but trails it on ZebraLogic and several instruction-following and multilingual evaluations. These results qualify the prose claim of text capability parity.

Omni-Thinking improves on Omni-Instruct across several shared tasks, but its AIME25 and function-calling scores remain below those of the same-size text-only Thinking model. WritingBench is one evaluation where Omni-Thinking exceeds that comparator.

5.1.2 Performance of Audio→Text

Audio→Text evaluation covers transcription, speech-to-text translation (S2TT), translating spoken input into written output, voice interaction, audio reasoning, and music understanding. Instruct achieves low word error rate (WER), the transcription error metric for which lower is better, on English, Chinese, and lyric transcription, but GPT-4o-Transcribe leads the 19-language Fleurs average and Gemini-2.5-Pro leads the listed translation directions. Music results favor Instruct over the reported generalist and specialist baselines.

• Librispeech clean/other WER is 1.22/2.48 for Omni-Instruct versus 1.39/3.75 for GPT-4o-Transcribe; Opencpop-test, a singing-transcription evaluation, gives WER of 1.54 versus 2.84 for Qwen2.5-Omni.
• The 19-language Fleurs average WER is 5.33 versus 4.48 for GPT-4o-Transcribe. Translation uses BLEU, a reference-based text-overlap score: Fleurs-en2xx, English speech translated into the other evaluated languages, is 37.50 versus 39.25 for Gemini-2.5-Pro.
• Table 7 gives VoiceBench, a voice-assistant evaluation suite, overall scores of 88.8 for 30B-A3B-Thinking and 89.5 for Flash-Thinking, versus 89.6 for Gemini-2.5-Pro. The surrounding prose assigns 89.5 to Thinking without this distinction.
• Omni-Instruct scores 77.5 on MMAU-v05.15.25, a multitask audio understanding and reasoning benchmark, versus 77.4 for Gemini-2.5-Pro, but 69.0 on the separate MMSU spoken-language reasoning row versus 77.7. Music scores include 52.0 on RUL-MuchoMusic, a music understanding evaluation, and 93.0 accuracy on GTZAN, a genre-classification dataset; multilabel music evaluation uses micro F1, an aggregate precision–recall measure, because generated labels lack calibrated ranking scores.

Strong English, Chinese, and lyric transcription coexist with weaker rankings on multilingual transcription and translation. The table uses lower-is-better WER for transcription and higher-is-better BLEU for translation; Seed-ASR also outperforms 30B-A3B-Instruct on both Wenetspeech subsets.

The table distinguishes 30B-A3B-Thinking’s VoiceBench overall score of 88.8 from Flash-Thinking’s 89.5. Separate audio-reasoning rows show that Instruct’s slight MMAU advantage over Gemini-2.5-Pro does not extend to MMSU.

Both Instruct variants exceed the listed baselines on every music evaluation, including RUL-MuchoMusic, GTZAN, and the multilabel datasets. The specialist comparisons use accuracy or micro F1 rather than ranking metrics requiring calibrated scores.

5.1.3 Performance of Vision → Text

Instruct scores 75.9 on MathVista mini and 56.3 on MATH-Vision full, two visual mathematics evaluations, compared with 74.8 and 38.1 for Qwen2.5-VL-72B. Thinking increases these scores to 80.0 and 62.9, but its long-video results trail Gemini-2.5-Flash-Thinking. The authors identify restricted context length and limited positional extrapolation, handling positions beyond the training range, as architectural limitations.

• Omni-Thinking scores 69.7 on Video-MME without subtitles, 49.0 on LVBench, and 72.9 on MLVU, three video-understanding benchmarks, versus 79.6, 64.5, and 82.1 for Gemini-2.5-Flash-Thinking.
• Thinking also improves HallusionBench, an evaluation of visual hallucination and illusion, from 59.7 to 62.8 and ChartQA, chart question answering, test average from 86.8 to 89.5. Its video scores are lower than Instruct’s on all three listed benchmarks.

Omni-Instruct’s strongest comparative results occur in visual mathematics and selected hallucination and video-understanding tests. Document understanding and counting remain below Qwen2.5-VL-72B on the listed rows.

Thinking gives competitive image-reasoning scores, including 80.0 on MathVista mini, but trails Gemini-2.5-Flash-Thinking on all three listed video evaluations. These results support the report’s stated long-video limitation.

5.1.4 Performance of AudioVisual Video→Text

Audiovisual evaluation tests whether the model can combine auditory and visual signals. Omni-Instruct scores 54.0 on WorldSense, an audiovisual understanding benchmark, versus 50.9 for Gemini-2.5-Flash and 45.4 for Qwen2.5-Omni. Omni-Thinking scores 75.8 on DailyOmni and 57.3 on VideoHolmes, two audiovisual reasoning benchmarks, exceeding Gemini-2.5-Flash-Thinking at 72.7 and 49.5 and the listed prior open-source results at 69.8 and 55.6.

WorldSense increases from 45.4 for Qwen2.5-Omni to 54.0 for Omni-Instruct, exceeding Gemini-2.5-Flash at 50.9. This benchmark supports improved joint auditory and visual understanding.

The released Thinking model exceeds both the listed previous open-source results and Gemini-2.5-Flash-Thinking on DailyOmni and VideoHolmes. Flash-Thinking slightly improves DailyOmni and ties VideoHolmes.

5.2.1 Evaluation of Zero-Shot Speech Generation

Evaluation of X→Speech primarily tests text-conditioned synthesis because the authors report a lack of relevant assessments for broader multimodal speech generation. On the SEED zero-shot test, generating a target voice without speaker-specific training for that test, Qwen3-Omni-30B-A3B achieves Chinese/English content-consistency WER of 1.07/1.39. English is best among the listed systems, while CosyVoice3 leads Chinese at 0.71; Table 13 reports only content consistency despite the evaluation introduction also mentioning speaker similarity.

5.2.2 Evaluation of Multilingual Speech Generation

The MiniMax multilingual test evaluates content consistency and speaker similarity, how closely synthesized speech matches the target voice, across all 10 output languages. Qwen3-Omni leads content consistency for Chinese, English, Italian, Korean, and French, and speaker similarity for English, German, Italian, Korean, and French. The remaining languages show mixed rankings.

• English content-consistency WER is 1.069 versus 2.164 for MiniMax and 2.339 for ElevenLabs; speaker similarity is 0.773 versus 0.756 and 0.613.
• Korean WER is 1.670 versus 1.747 and 1.865, with speaker similarity 0.778 versus 0.776 and 0.700.
• Spanish WER is 1.765 versus MiniMax’s 1.029, illustrating the language dependence of performance.

Content consistency and voice similarity have different winners across languages. Qwen3-Omni leads both metrics for English, Italian, Korean, and French, while MiniMax leads both for Spanish and Japanese.

5.2.3 Evaluation of Cross-Lingual Speech Generation

Cross-lingual voice cloning, preserving a reference voice while generating another language, is evaluated on the CosyVoice3 cross-lingual test set. Qwen3-Omni has better content consistency than CosyVoice3 for every listed direction into English and Korean, but worse scores for every listed direction into Japanese. The Japanese comparison uses different preprocessing: CosyVoice3 converts characters to phonetic kana, while Qwen3-Omni uses no text normalization.

• zh-to-en is 2.76 versus 2.98 for CosyVoice3; zh-to-ko is 5.13 versus 14.4.
• zh-to-ja is 8.29 versus 7.08, en-to-ja is 7.53 versus 6.80, and ko-to-ja is 4.24 versus 3.93.

Qwen3-Omni leads all listed directions into English and Korean, while CosyVoice3 leads all directions into Japanese. The reported preprocessing difference limits comparability for Japanese output.

6 Evaluating Non-Degradation Across Modalities

The controlled non-degradation study compares Qwen3-30B-A3B-Base-202507, Qwen3-VL-30B-A3B-Base-202507, and Qwen3-Omni-30B-A3B-Base-202507. The report states that parameter scale, training schedules, shared modality datasets, batch sizes, effective modality-specific epochs, and training compute are matched, with supplementary audio and audiovisual data distinguishing Omni. Table 16 supports broadly preserved capabilities and several visual gains, while also recording declines on individual text, image, and video tasks.

• Against the text baseline, Omni scores 81.69 versus 81.24 on MMLU, a general knowledge benchmark, and 73.96 versus 69.70 on EvalPlus, a code-generation evaluation, but 60.42 versus 60.84 on MATH, mathematics problems, and 64.79 versus 65.75 on MultiPL-E, multilingual code generation.
• Against the vision-language baseline, Omni scores 59.33 versus 57.22 on MMMU validation, multidisciplinary visual questions, 83.31 versus 81.17 on InfoVQA test, infographic question answering, and 51.07 versus 48.61 on LVBench.
• RealWorldQA average, real-world visual question answering, declines from 73.98 to 71.89 and MVBench, a video-understanding evaluation, from 71.87 to 69.50. No comprehensive model-scale sweep or uncertainty estimates are provided, so the study does not establish universal or statistical non-degradation.

The matched base-model study records gains such as MMMU validation at 59.33 versus 57.22 and InfoVQA test at 83.31 versus 81.17, together with losses such as MVBench at 69.50 versus 71.87. Its mixed task-level results support broad preservation at the tested scale without proving that every capability is unchanged.

7 Conclusion

The conclusion presents integrated multimodal training and frame-level speech streaming as the report’s principal contributions. It identifies multi-speaker ASR, video optical character recognition (OCR), reading text from video, audiovisual proactive learning, and improved agent workflows and function calling as future work. The abstract states that Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner are publicly released under Apache 2.0.

• The paper lists https://github.com/QwenLM/Qwen3-Omni, https://huggingface.co/spaces/Qwen/Qwen3-Omni-Demo, and https://modelscope.cn/studios/Qwen/Qwen3-Omni-Demo.
• The conclusion’s phrase “interactive speech support in over 20 languages” is inconsistent with the explicit counts of 19 speech-input and 10 speech-output languages in Table 3.

8 Authors

The report is authored by Qwen Team and lists Core Contributors and Contributors separately; the Contributors list is explicitly marked as alphabetical. Core Contributors begin with Jin Xu, Zhifang Guo, Hangrui Hu, and Yunfei Chu and end with Jingren Zhou and Junyang Lin. No institutional affiliation or conference or journal venue is explicitly provided.

Appendix

• 9.1 More Evaluation on Speech and Music Understanding reports weaker transcription and music results for Thinking than Instruct: Librispeech clean/other WER rises from 1.22/2.48 to 2.22/4.38, the 19-language Fleurs average rises from 5.33 to 8.63, and RUL-MuchoMusic falls from 52.0 to 48.3. The authors suggest that reasoning may increase hallucination on these tasks, but do not directly quantify that mechanism. Translation results are not uniformly worse: Fleurs-xx2zh improves from 33.13 to 34.51 BLEU.
• 9.2 Qualitative Results from Qwen3-Omni-30B-A3B-Captioner provides three examples: Analysis of Expressive Speech, Analysis of Complex Scene Sound Effect, and Analysis of Mixed Speech, Audio, and Music. Generated descriptions include dialogue, vocal delivery, acoustic texture, and temporal events; the complex-scene caption describes a 25-second soundscape with events around 9, 15, and 19 seconds. These outputs illustrate descriptive breadth but include inferred settings and relationships, without reference captions or a quantitative accuracy assessment.

Brief Thoughts

The controlled base-model comparison in Table 16 and the audiovisual results in Tables 11–12 support useful multimodal integration at the tested scale. Deployment assessment should distinguish theoretical startup latency from sustained generation speed and compare Instruct with Thinking on the intended task, since reasoning improves some evaluations while weakening transcription and music perception. Non-degradation is supported as broad capability preservation under the reported conditions, with task-level declines and no comprehensive scale sweep.