FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

arXiv:2608.24168 · cs.CL, cs.SD · Submitted 2026-08-25 · Read on arXiv

cs.CL, cs.SD

Submitted: 2026-08-25

Updated: 2026-08-27

Comments: 20 pages, 3 figures. In this revision, the author list is ordered alphabetically by given name and an author-contribution statement is added; the technical content is unchanged

Code: https://github.com/FireRedTeam/FireRedAudio

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 19× in this paper · explained in 1 episode
• Alignment — 13× in this paper · explained in 1 episode
• Benchmark — 9× in this paper · explained in 2 episodes
• Transformer — 9× in this paper · explained in 2 episodes
• Grounding — 7× in this paper · explained in 3 episodes
• Abstraction — 4× in this paper · explained in 1 episode
• Adaptation — 4× in this paper · explained in 1 episode
• Latent Space — 4× in this paper · explained in 7 episodes
• Distillation — 3× in this paper · explained in 2 episodes
• Embeddings — 3× in this paper · explained in 2 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Regularization — 2× in this paper · explained in 3 episodes
• Acoustic-Parameter Specification — 1× in this paper
• Aggregation — 1× in this paper · explained in 1 episode
• Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 4 episodes
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Descriptive-Style Directive — 1× in this paper
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Flow Matching — 1× in this paper · explained in 8 episodes
• Generative Modeling — 1× in this paper · explained in 3 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Latent Representation — 1× in this paper · explained in 2 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Tokenization — 1× in this paper · explained in 2 episodes
• Variational Autoencoder (VAE) — 1× in this paper · explained in 3 episodes
• Variational Bayes — 1× in this paper · explained in 2 episodes

Sources

• F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
• LongCat-Next: Lexicalizing Modalities as Discrete Tokens
• UALM: Unified Audio Language Model for Understanding, Generation and Reasoning
• Unified Audio Intelligence Without Regressing on Text Intelligence
• Kimi-Audio Technical Report
• Qwen3.5-Omni Technical Report
• Qwen3-Omni Technical Report
• UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization
• Multimodal Latent Language Modeling with Next-Token Diffusion
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
• VibeVoice Technical Report
• Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation
• Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation
• Auto-Encoding Variational Bayes
• Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
• UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
• Flow Matching for Generative Modeling
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering