Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token. This creates an ‘information void’ where semantic information that could be inferred from unmasked tokens is lost between denoising steps. We introduce Continuously Augmented Discrete Diffusion (CADD), a framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than collapsed ‘information voids’. At… Read More
Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token. This creates an ‘information void’ where semantic information that could be inferred from unmasked tokens is lost between denoising steps. We introduce Continuously Augmented Discrete Diffusion (CADD), a framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than collapsed ‘information voids’. At…
Related Posts
A smarter way for large language models to think about hard problems
This new technique enables LLMs to dynamically adjust the amount of computation they use for…
5 Python Libraries for Advanced Time Series Forecasting
Predicting the future has always been the holy grail of analytics.
4 ways to refine your content in Flow
You’ll now get more creative control in Flow with new refinement and editing capabilities.