Monthly AI Papers: November 2025

Contents

Trends
Top Papers in November 2025
Sources

Trends

November 2025’s theme is making better use of model capacity: the three selected papers change image-prediction targets, language-model memory, or student-training feedback to improve generation under specific constraints. Just image Transformers (JiT), which map noisy image patches directly to clean patches, show that the quantity a network predicts matters when patch dimensionality is high. Kimi Linear combines recurrent memory, a fixed-size summary of token history, with periodic global attention over previous tokens. Generative Adversarial Distillation (GAD), which trains a student against a discriminator that scores teacher and student responses, supplies feedback on the student’s own generated answers using only teacher text. Together, these studies examine what information a model must represent and how training directs its limited capacity. 1, 2, 3

The reported gains require different engineering interpretations. JiT’s nearly unchanged computation across resolutions depends on increasing patch size with resolution, and its comparisons measure a single forward pass rather than total training cost. Kimi Linear separates faster prompt processing and single-request decoding from the larger acceleration enabled by memory savings and larger batches. GAD improves conversational judging scores under equal training epochs, but the paper does not compare computational cost with its supervised baseline. These results support evaluating each design against its actual constraint: image representation, long-context serving, or text-only teacher access.

Top Papers in November 2025

1. Back to Basics: Let Denoising Generative Models Denoise

https://arxiv.org/abs/2511.13720
Back to Basics: Let Denoising Generative Models Denoise | Summary
Back to Basics: Let Denoising Generative Models Denoise separates prediction space, the quantity directly emitted by the network, from loss space, the quantity used to compute training error. JiT directly predicts clean images with a Vision Transformer, a Transformer operating on image patches, without a learned latent tokenizer that encodes pixels for generation, external pre-training, or auxiliary losses. Its capacity argument rests on the manifold assumption: natural images approximately occupy a low-dimensional structure within pixel space, while noise-containing targets occupy the surrounding high-dimensional space.
On ImageNet 256×256, JiT-B/16, the base-size model with 16×16 patches, achieves FID-50K of 8.62 with clean-image prediction, versus 372.38 with noise prediction and 96.53 with flow-velocity prediction, which predicts the noisy sample’s rate of change. FID-50K is Fréchet Inception Distance measured on 50,000 generated images; lower is better. The comparison uses the same architecture, velocity-space loss, and 200 training epochs, with classifier-free guidance, which controls class conditioning during sampling. Guidance settings and exponential moving average decays, which control parameter averaging during training, are selected separately for each model.
Compression experiments support the capacity argument within the tested settings. A 64-dimensional input bottleneck, an intermediate representation narrower than the 768-dimensional raw patch, improves JiT-B/16 FID from 8.62 to 7.35; a 16-dimensional bottleneck still yields 9.40. In the synthetic experiment, clean-image prediction preserves a two-dimensional spiral embedded in 512 observed dimensions using 256-dimensional hidden layers. Noise and velocity prediction fail in that setting. These experiments do not directly measure natural images’ intrinsic dimensionality.

2. Kimi Linear: An Expressive, Efficient Attention Architecture

https://arxiv.org/abs/2510.26692
Kimi Linear: An Expressive, Efficient Attention Architecture | Summary
Kimi Linear: An Expressive, Efficient Attention Architecture introduces Kimi Delta Attention (KDA), a linear attention mechanism that compresses history into fixed-size memory and learns separate forgetting rates for individual feature channels. Its delta-rule update corrects stored key–value associations after memory decay. The architecture repeats three KDA layers followed by one Multi-Head Latent Attention (MLA) layer, the paper’s global-attention mechanism. Each KDA head stores a fixed 128×128 state; the complete hybrid retains growing attention-history caches in its MLA layers.
Parameter-matched comparisons use 48B total parameters, 3B activated parameters used per forward pass, and the same 1.4T-token training recipe. Kimi Linear raises MMLU-Pro, a multitask language-understanding benchmark, from MLA’s 47.2 to 51.0 and the average across eight 128k-context metrics from 52.2 to 54.5. Gains vary by task: MATH, a mathematics benchmark, ties MLA at 54.7, while MLA remains stronger on three of the eight long-context metrics. The results evaluate the complete hybrid architecture and do not attribute every gain to channel-wise forgetting alone.
The hybrid reduces key–value cache usage, the stored attention history, by up to 75%. At 1M context, the timing plots report 2.9× faster prompt processing and approximately 2.2× faster decoding at batch size 1 relative to MLA. A separate larger-batch comparison reports 1.84 ms versus 11.48 ms per generated token, labeled 6.3× acceleration, including the batching benefit enabled by reduced memory consumption. Accelerator hardware and exact larger-batch sizes are unspecified. The released 5.7T-token model’s comparison with Moonlight also changes total parameters from 16B to 48B, making the matched 1.4T-token runs stronger evidence for architectural comparison.

3. Black-Box On-Policy Distillation of Large Language Models

https://arxiv.org/abs/2511.10643
Black-Box On-Policy Distillation of Large Language Models | Summary
Black-Box On-Policy Distillation of Large Language Models extends knowledge distillation, transferring a teacher’s behavior to a student, to black-box teachers that expose generated text without internal probabilities. GAD enables on-policy learning, training from responses sampled from the current student, through an adaptive discriminator. The discriminator learns to rank teacher responses above student responses and supplies rewards for student optimization. Both models receive one warmup epoch before two adversarial training epochs, during which both continue to update.
Using 200K prompts paired with GPT-5-Chat responses, GAD improves all five tested students across four evaluation datasets relative to both their initial models and sequence-level knowledge distillation (SeqKD), which fits students to teacher responses. Under the same three-epoch schedule, Qwen2.5-14B-Instruct scores 52.1 on the LMSYS-Chat conversational test set, versus 50.6 for SeqKD and 51.7 for the teacher. These are averaged GPT-4o judging scores derived from each answer’s rating relative to a GPT-4o-generated reference answer, rather than accuracy percentages or direct win rates.
Human comparisons against SeqKD report win/tie/loss rates of 52%/40%/8% for Qwen2.5-7B-Instruct, 68%/8%/24% for Qwen2.5-14B-Instruct, and 44%/40%/16% for Llama-3.1-8B-Instruct. A frozen-discriminator experiment exhibits reward hacking, exploiting the reward signal through undesirable outputs, after around 300 steps, with responses reaching about 1300 tokens; the adaptive version maintains comparatively stable lengths in the displayed run. Main scores lack uncertainty estimates, checkpoint selection lacks a clearly separate split, and GAD responses are longer than SeqKD responses. The reported training time of about 30 hours on 16 H100 GPUs for the 14B student provides a cost reference, without establishing cost parity with SeqKD.