A Survey on Continuous Diffusion and Flow Language Models

Sophia Tang¹ Chanhyuk Lee² Floor Eijkelboom³ Jinwoo Kim²

sophtang@engineering.upenn.edu chanhyuk-lee@kaist.ac.kr f.eijkelboom@uva.nl jinwoo-kim@kaist.ac.kr

[GitHub Collection] [Explore the Map] [Read Survey]

Advances in continuous language models (top) and in foundations of continuous generation of images and video (bottom). Hover or tap a paper for links, or select a legend entry to highlight a family. Dashed arrows mark direct adaptations to language.

Abstract

Continuous diffusion and flow language models generate text by transforming noise into continuous representations of tokens.

These models refine many positions in parallel and keep the dependencies between them, and gradients can steer them toward a goal. Recent models match discrete diffusion on language benchmarks and reach billions of parameters.

We explain how these models represent text, learn generative dynamics, and generate in few steps. We then review scaling, applications, evaluation, and open questions.

Read the full abstract

Continuous diffusion and flow language models generate text by transforming noise into continuous representations of tokens. They can refine many positions in parallel while preserving dependencies between them, and their continuous states allow generation to be steered using gradients of a differentiable objective. Joint generation is especially useful in the few-step regime, where standard parallel discrete-diffusion samplers can incur a factorization error by sampling multiple tokens conditionally independently. Recent advances have made continuous models competitive with discrete diffusion on language benchmarks and have enabled scaling to billions of parameters. This survey explains the principles behind these advances and organizes the field by how models represent text, learn generative dynamics, and generate samples. We examine the methods that enable few-step generation, including flow maps and distillation, and connect them to developments in image and video generation. We then review scaling, applications, and evaluation, and identify open questions in representation learning, scaling, few-step distillation, post-training, multimodality, and safety.

Survey Contents

Select a section to show its subsections. Hover a section with a figure to preview it.

1Introduction
2Motivations and Preliminaries
1. 2.1Discrete Diffusion and Masked Generation
2. 2.2Factorization Error in Parallel Decoding
3. 2.3Difficulty of Controllable Generation
4. 2.4Continuous Processes for Discrete Data
3Continuous Language Models
1. 3.1Continuous Representation of Language
  1. 3.1.1One-Hot and Binary Token Encodings
  2. 3.1.2Vocabulary Embeddings
  3. 3.1.3Contextual Token Embeddings
  4. 3.1.4Compressed Latent Representations
  5. 3.1.5Simplex Representations
  6. 3.1.6Spherical Representations
  7. 3.1.7What Makes a Good Representation?
2. 3.2Parameterization and Objectives
  1. 3.2.1Prediction Parameterizations
  2. 3.2.2Cross-Entropy Objectives
  3. 3.2.3Regression Objectives
  4. 3.2.4Likelihood Objectives
3. 3.3Noise Scheduling and Time Reparameterization
4. 3.4Self-Conditioning
5. 3.5Architecture
6. 3.6Commitment and Ordering
4Inference Strategies
1. 4.1Sampling and Time Allocation
2. 4.2Guidance and Control
3. 4.3Self-Conditioning and Iterative Refinement
4. 4.4Commitment and Ordering
5. 4.5Decoding and Termination
6. 4.6Inference Efficiency
5Few-Step Generative Modeling
1. 5.1Formulations and Generative Interfaces
  1. 5.1.1Reducing Solver Error
  2. 5.1.2Flow Maps
  3. 5.1.3Two-Time Denoisers
  4. 5.1.4Pushforward Maps
  5. 5.1.5Autonomous Endpoint Maps
2. 5.2Learning Objectives and Training Strategies
  1. 5.2.1Training Flow Maps
  2. 5.2.2Distillation and Self-Distillation
  3. 5.2.3Training Pushforward Maps
  4. 5.2.4Training Autonomous Endpoint Maps
  5. 5.2.5Noise-Data Coupling
3. 5.3Sampling with Learned Maps
4. 5.4Guided Few-Step Generation
  1. 5.4.1Distilling Guided Flow into Flow Map
  2. 5.4.2Guiding the Flow Map
5. 5.5Few-Step Generation with Iterative Feedback
  1. 5.5.1Self-Conditioning in Flow Maps
  2. 5.5.2Commitment and Ordering in Flow Maps
6Scaling Continuous Language Models
1. 6.1Early Attempts at Scaling
2. 6.2Recent Progress
  1. 6.2.1Training from Scratch
  2. 6.2.2Adapting Pretrained Models
7Applications and Evaluation
1. 7.1Unconditional Language Modeling
  1. 7.1.1Tasks and Datasets
  2. 7.1.2Evaluation Metrics
2. 7.2Conditional Language Modeling
  1. 7.2.1Tasks and Evaluation for Small Models
  2. 7.2.2Tasks and Evaluation for Large Models
3. 7.3Reasoning for Puzzle Solving
  1. 7.3.1Single-Solution Tasks
  2. 7.3.2Multi-Solution Tasks
4. 7.4Biological and Scientific Applications
  1. 7.4.1Tasks and Datasets
  2. 7.4.2Conditional Generation and Reward Guidance
  3. 7.4.3Evaluation Metrics
8Challenges and Future Directions
1. 8.1Scalable Language Representations
2. 8.2Scaling Pretraining and Few-Step Distillation
3. 8.3Post-training
4. 8.4Multimodality
5. 8.5Safety
9Conclusion

News and Updates

Nov 9, 2026
The survey is live on alphaXiv.
Nov 9, 2026
Our Map of Continuous Language Models and paper list are live.

BibTeX

If you find this survey useful, please cite it.

@article{tang2026survey,
  author = {Tang, Sophia and Lee, Chanhyuk and Eijkelboom, Floor and Kim, Jinwoo},
  title = {A Survey on Continuous Diffusion and Flow Language Models},
  url = {https://www.alphaxiv.org/abs/2611.continuous-diffusion-flow-language-models},
  publisher = {alphaXiv},
  year = {2026}
}