Summary explanation: AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
08 Sep 2026 | Paper Review Speech Generation Speech Editing Flow Matching Reinforcement LearningContents
- Summary
- 1 Introduction
- 2 Data Construction
- 3 Model Design
- 4 Model Training
- 5 Model Acceleration
- 6 Model Inference
- 7 Performance
- 8 Conclusion
- Contribution
- Appendix
- Brief Thoughts
This article explains the key points of AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing.
- 2026-09-08 (arXiv)
- Ma, Ziyang, Niu, Zhikang, Tu, Wenming, Wang, Tianrui, Yan, Ruiqi, Liu, Junxi, Huo, Yanru, Huang, Nickk, Liu, Yang, Xie, Qicong, et al.
- Paper
- Github
- Project Page
Summary
- AuK is an open-source foundational model that maps a natural-language instruction and optional audio context to a target waveform, unifying speech generation, editing, enhancement, and separation. Its corpus contains approximately 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision; the distilled AuK-Flash runs four CFG-free steps and is reported to provide a 4.5× wall-clock speedup under matched conditions.
The overview compares AuK, AuK-Flash, and prior systems across selected generation, editing, enhancement, and separation metrics. It shows that the full and distilled variants have task-dependent strengths rather than uniformly identical performance.
1 Introduction
The paper frames practical speech requests as combinations of synthesis, local editing, attribute preservation, restoration, and source selection. A unified model must support both text-only and audio-conditioned inputs while accommodating heterogeneous objectives and subjective editing quality.
The figure illustrates the common interface of natural-language instructions with optional speech input and output audio. It defines the five task families covered by the model.
2 Data Construction
Pre-training uses a common instruction–optional-input-audio–target-waveform interface across speech generation, acoustic editing, paralinguistic editing, content editing, and enhancement and separation. This organization is intended to let a shared backbone learn both creation and reference-conditioned transformation.
Speech Generation
For transcript-free zero-shot TTS, the authors form bidirectional prompt–target pairs from different utterances by the same speaker, supplying prompt audio and target text but not the prompt transcript. Instruct TTS uses Qwen3-Omni-generated free-form descriptions and structured voice attributes without reference audio.
Acoustic Editing
Acoustic-editing supervision modifies speaking rate, loudness, or pitch while preserving content and speaker identity. Targets use rate multipliers of 0.5×, 0.75×, 1.25×, 1.5×, and 2.0×; loudness offsets of ±5, ±10, and ±15 dB; and pitch shifts of ±1, ±2, and ±3 semitones.
2.3 Paralinguistic Editing
Paralinguistic data cover emotion, timbre, de-accenting, nonverbal vocalizations, and whisper conversion. The construction includes eight emotion categories, 13 Chinese dialect or regional-accent categories, and 39 normalized nonverbal-event types.
Content Editing
Content-editing pairs support insertion, deletion, and substitution in spoken speech and lyrics. Word-level alignment identifies local masked regions, and ASR-based quality control retains synthesized samples with low word error rate; lyric replacements preserve Chinese character counts or English word counts.
Enhancement and Separation
Enhancement and separation data apply noise, reverberation, channel degradation, and multi-speaker or music mixtures. Targets may remove only the corruption or source named by the instruction rather than always restoring fully clean speech.
The radial layout summarizes the corpus taxonomy and representative instruction-level operations. Its sector widths are explicitly not data-volume indicators.
3 Model Design
AuK combines a frozen multimodal language model, an audio VAE, and a FLUX-style rectified-flow Transformer. Semantic features encode instructions and audio context, while VAE latents provide acoustic conditioning and waveform reconstruction.
Overall Architecture
Reference audio, when available, is encoded by both the MLLM audio encoder and the VAE; text-only tasks omit those branches. Dual-stream MMDiT blocks exchange information between semantic and acoustic streams before single-stream DiT blocks refine their concatenation.
3.2 MLLM Semantic Condition
Qwen2.5-Omni supplies the semantic condition by encoding the instruction and, when present, reference audio. AuK learns an unconstrained weighted sum of layer-normalized hidden states rather than choosing a fixed MLLM layer.
3.3 VAE Acoustic Condition and Reconstruction
AuK-VAE maps 24 kHz audio to 64-dimensional latents at 50 Hz and uses a causal decoder for waveform generation. A normalizing flow regularizes the posterior during VAE training but is not used for acoustic conditioning or reconstruction at generation time.
Transformer Backbone
The approximately 1.5-billion-parameter backbone has 30 layers: 10 dual-stream MMDiT blocks followed by 20 single-stream DiT blocks, with hidden dimension 1536 and 24 attention heads. The final DiT stack predicts rectified-flow velocity at target-latent positions.
The diagram shows semantic conditioning from the MLLM, optional VAE reference-audio conditioning, dual-stream MMDiT fusion, single-stream DiT refinement, and VAE decoding. It clarifies how the same architecture supports text-only generation and audio-conditioned editing.
4 Model Training
Training comprises VAE training, a generation-only warm-up followed by joint generation–editing pre-training, and two post-training stages. The MLLM encoder and VAE remain frozen during unified pre-training and post-training.
VAE Training
AuK-VAE is trained for 1.24 million updates on approximately 3 million hours of speech, music, and general audio sampled at a 6:3:1 instance ratio. Its objective combines multi-scale log-mel reconstruction, adversarial and feature-matching losses, and KL regularization with λmel = 15, λadv = 1, λfeat = 2, and λKL = 5.
4.2 Unified Pre-Training
Unified pre-training uses 50k generation-only updates followed by 600k joint updates across all five task families. Second-stage sampling probabilities are 28.10% generation, 23.02% content editing, 23.17% enhancement and separation, 21.75% paralinguistic editing, and 3.96% acoustic editing.
The table defines the fixed second-stage joint-training mixture. Acoustic editing receives substantially less sampling probability than the other four families.
4.3 Post-Training
Post-training separates open-ended editing alignment from speech-generation optimization. Editing uses human preference optimization because the authors did not identify a sufficiently broad preference dataset or ready-made reward model, whereas generation uses automatic content, speaker-similarity, and style-consistency rewards.
Editing Preference Optimization
Annotators rate 10 or 20 editing candidates per instruction as failure, partial completion, or success. After filtering, the preference set contains 818 informative groups and 9,080 rated candidates; a label-smoothed ordinal listwise objective is trained for 104 optimizer updates.
4.3.2 Generation Reinforcement Learning
Generation RL uses Flow-GRPO on 10,000 zero-shot prompts and 5,000 instruction-following prompts, with G = 16 candidates and 500 optimizer updates. Speaker reward is standardized only among candidates with tolerant recognition error Ei = 0, preventing high similarity from compensating for incorrect content.
5 Model Acceleration
AuK-Flash is produced through consistency initialization followed by task-routed Decoupled DMD. Because uniform DMD degraded multi-speaker and vocal separation, separation examples instead use supervised clean-latent regression and are excluded from DMD and fake-score updates.
6 Model Inference
Inference optionally applies a Prompt Enhancer before conditional flow sampling in VAE latent space. Full AuK uses 32 Euler function evaluations and CFG scale 2.0, while AuK-Flash uses four evaluations without CFG.
Prompt Enhancer
The Prompt Enhancer identifies the task, extracts and validates arguments, rewrites requests toward supported templates, prepares optional audio, and estimates duration. It maps continuous descriptions of rate, loudness, and pitch to training-supported discrete values and rejects unsupported requests.
Inference Strategy and Configuration
Input audio is converted to mono, resampled to 24 kHz, and encoded as semantic and acoustic context; text-only generation uses the instruction alone. The 4.5× Flash speedup is measured against 32-NFE, CFG-2.0 AuK with matched hardware, output duration, and batch size.
The table isolates the computational difference between full AuK and Flash: four rather than 32 evaluations and no CFG, with the same latent representation and decoder. The stated speedup pertains to this sampler configuration.
7 Performance
Evaluation covers VAE reconstruction, zero-shot and instruction-controlled generation, instruction-guided editing, enhancement, separation, and super-resolution. The reported results distinguish full-model linguistic and edit-fidelity strengths from Flash advantages on several perceptual metrics.
This is the principal cross-task comparison. It documents strong full-model results on generation and editing while showing Flash advantages on selected all-rubric or perceptual-quality measures.
VAE Reconstruction Results
Across Seed-TTS-Eval speech, AudioSet general audio, and MUSDB18-HQ music, AuK-VAE is reported best on PESQ, STOI, Mel Dist, and STFT. On speech reconstruction it reaches PESQ 4.143, STOI 0.982, Mel Dist 0.574, and STFT 1.457.
The reconstruction comparison evaluates the VAE substrate used for generation. AuK-VAE is reported best on all four metrics across speech, general audio, and music.
7.2 Generation Ability
On Seed-TTS-Eval, AuK reports average recognition error of 2.65% and SIM of 0.795, versus 3.07% and 0.745 for Qwen3-TTS; AuK-Flash reports 2.85% and 0.790. On InstructTTSEval, AuK reaches Chinese DSD 83.37%, while Flash reaches English DSD 82.40%, tying Qwen3-TTS-VD in the summarized comparison.
7.3 Editing Ability
Editing evaluation combines instruction following and preservation with operation-level semantic and acoustic tests. The evaluation distinguishes average metrics from all-rubric success and separates content preservation from perceptual and speaker-preservation measures.
7.3.1 General Speech Editing
On MMAE-Speech, AuK attains IFR 48.23% and CR 88.11%, while AuK-Flash attains the highest EMR at 13.85%. On SpeechEditBench, AuK leads the displayed dedicated models in content, emotion, prosody, and acoustic editing, while Flash leads paralinguistic editing at 39.25%.
Speech Enhancement and Separation
On DNS Challenge, full AuK has the lowest dWER at 2.66% and both variants reach SIM 0.99; Flash has the highest UTMOS at 4.05. On Libri2Mix, AuK has lower WER/PER of 9.12%/6.63%, while Flash has higher OVRL/UTMOS of 3.32/4.03.
Discovery
Cross-utterance prompt–target pairing is proposed to emphasize speaker-invariant characteristics and enable transcript-free voice cloning. The authors also qualitatively observe transfer from whisper editing to text-driven whisper generation and from Chinese de-accenting supervision to some English accents, while noting that these observations are not comprehensive evaluation.
8 Conclusion
The paper concludes that a shared MLLM–VAE–rectified-flow architecture, five-family supervision, complementary post-training, and task-routed distillation support a broad speech interface. It acknowledges that reliable handling of unconstrained free-form requests still benefits from explicit task routing and prompt enhancement.
Contribution
The report lists Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, and Yanru Huo as core contributors. Xie Chen is identified as the corresponding author.
The table separates instruction-following TTS into APS, DSD, and role-playing measures for Chinese and English. It supports the distinction between full AuK's Chinese scores and Flash's English DSD tie.
The MMAE-Speech table distinguishes instruction following and preservation from exact all-rubric success. AuK leads IFR and CR, whereas Flash leads EMR.
The Basic split decomposes semantic editing into deletion, insertion, and substitution in Chinese and English. It includes no-edit WER as a preservation check.
The Full split provides the more difficult semantic-editing comparison. Both AuK variants substantially reduce average WER relative to Ming-UniAudio, with operation-dependent differences between them.
The acoustic-editing table evaluates speaking-rate, pitch, and volume control alongside intelligibility, similarity, and control-error measures. The variants trade off WER and speaker-similarity results across operations.
On DNS Challenge 2020, full AuK has the lowest dWER and both variants reach SIM 0.99, while Flash has the highest UTMOS. Perceptual and recognition-oriented metrics therefore select different variants.
The CHiME-4 comparison reports Flash with the best displayed WER and UTMOS. Its OVRL of 3.35 ties the best result in the complete table.
For two-speaker separation, full AuK yields lower WER and PER, while Flash has higher OVRL and UTMOS. Both variants match the leading SIM of 0.96.
The super-resolution results show Flash leading six of seven reported metrics, including OVRL, UTMOS, WER, and PER, while full AuK reaches the highest SIM. Scores average eight degradation subsets, including an out-of-range 2-kHz condition.
Appendix
- The appendix expands benchmark evidence with broader generation baselines and operation-level editing, enhancement, separation, and super-resolution results. It specifies that MMAE-Speech uses the Prompt Enhancer and reports detailed results for deletion, insertion, substitution, speed, pitch, volume, DNS Challenge, CHiME-4, Libri2Mix, and VCTK-SR.
Brief Thoughts
The report provides unusually broad quantitative coverage across generation, editing, enhancement, separation, and super-resolution, including explicit full-model versus distilled-model trade-offs. Its main practical limitation is that robust free-form usability still depends on an external Prompt Enhancer and task routing.