Monthly AI Papers: February 2025
28 Feb 2025 | Paper Review Monthly Papers Test-Time Scaling Knowledge Distillation QuantizationContents
Trends
February 2025’s theme is resource allocation for language models: the three selected papers examine how reasoning computation, teacher supervision, and weight precision affect quality under different budgets. s1 studies test-time scaling, spending more computation while answering; Distillation Scaling Laws studies knowledge distillation, training a student on a teacher’s output probabilities; and Matryoshka Quantization studies quantization, representing weights with fewer discrete values. Together, they make the resource being controlled explicit, while measuring different outcomes: reasoning accuracy, validation loss, and accuracy at different weight-storage budgets. 1, 2, 3
Teacher supervision connects the training results, but the form of supervision matters. s1 adapts a pretrained model using 1,000 selected questions with teacher-generated reasoning traces, intermediate reasoning before an answer. Distillation Scaling Laws instead fits student performance under training on full teacher probability distributions and finds a capacity gap, a mismatch between teacher and student learning abilities that can make a stronger teacher produce a worse student. Its fitted law remains unverified for training on generated sequences, so it does not directly explain s1’s results. MatQuant also tests co-distillation, using higher-precision outputs to supervise lower-precision models, with gains that depend on the configuration.
The engineering lesson is to compare quality alongside complete resource accounting. s1’s gains from extending reasoning eventually plateau, and its 1,000-example training count excludes pretraining and teacher-data production costs. The distillation study’s compute-optimal comparisons, which minimize predicted loss under a fixed computation budget, favor direct supervised training when a teacher’s full training cost is charged to one student. MatQuant demonstrates accuracy improvements at low weight precision, but supplies no measured latency or throughput gains. These results support testing specific operating points; they do not establish a common measure of end-to-end efficiency.
Top Papers in February 2025
1. s1: Simple test-time scaling
- https://arxiv.org/abs/2501.19393
- s1: Simple test-time scaling | Summary
- s1: Simple test-time scaling combines s1K, 1,000 examples selected for quality, difficulty, and diversity, with supervised fine-tuning, training on supplied reasoning traces and answers. Adapting Qwen2.5-32B-Instruct takes 26 minutes on 16 NVIDIA H100 graphics processing units. Budget forcing then controls stopping: inserting an end-of-thinking marker shortens reasoning, while suppressing that marker and appending “Wait” extends it.
- On AIME24, the 30-problem 2024 American Invitational Mathematics Examination benchmark, accuracy rises from the base model’s 26.7% to 50.0% after fine-tuning and 56.7% with budget forcing. This separates the training gain from the additional inference intervention. With a shared maximum of around 30,000 thinking tokens, the units generated during reasoning, the curated subset scores 50.0% versus 53.3% for the full 59K-example pool; training requires approximately 7 versus 394 H100 GPU hours.
- Longer reasoning helps only within tested limits: repeated stopping suppression eventually plateaus and can induce repetitive loops or exceed the model’s maximum sequence length. The comparison with majority voting, choosing the most frequent answer across independent attempts, changes both training and inference method, so it does not isolate sequential versus parallel computation. The full-pool comparison’s uncertainty interval includes gains and losses, supporting competitiveness of s1K without establishing statistical equivalence. The s1.1 appendix results were added in March 2025 and are separate from the original experiments.
Accuracy rises with average thinking-token use on all three benchmarks over the displayed settings. The finite range demonstrates tested operating points without establishing continued improvement beyond them.
The domain display shows the breadth of the selected examples, while the comparison places s1 near models trained on much larger reasoning datasets at similar MATH500 accuracy. The example counts exclude pretraining and teacher-data production costs.
2. Distillation Scaling Laws
- https://arxiv.org/abs/2502.08606
- Distillation Scaling Laws | Summary
- Distillation Scaling Laws predicts student validation cross-entropy, the average negative log probability assigned to observed tokens, from student parameter count, distillation-token count, and teacher validation loss. The empirical law fits 697 distillation runs with reported relative prediction error of approximately 1% or less, including tested extrapolations from weaker to stronger models. Experiments use English C4 text and models ranging from 143M to 12.6B non-embedding parameters, excluding token-embedding weights, with training runs extending to 512B tokens.
- Teacher quality has a student-dependent optimum. At fixed student size and token budget, improving teacher validation loss first reduces student loss and can then increase it, demonstrating the capacity gap. When teacher costs are excluded, a suitable teacher can beat direct supervised next-token training under modest student budgets. Charging teacher inference introduces a further trade-off: a larger teacher can improve supervision while leaving fewer affordable student-training tokens.
- Charging the full cost of teacher training to one student favors direct supervised training in the paper’s fitted compute-optimal comparisons. Reusing a teacher or sharing its costs can change that accounting, while sufficiently large student budgets let supervised training match optimally selected distillation. The coefficients apply to the tested architecture and dataset; generated-sequence distillation and reasoning performance remain outside the demonstrated scope. Numerical optima far beyond observed training ranges are predictions from the fit.
Predictions fitted to students with L_S > 2.3 track lower-loss students as well as unseen teacher configurations. The curved relationships show that improving teacher loss helps only up to a student-dependent optimum, linking the extrapolation test to capacity-gap behavior.
The fixed-compute profiles show why parameter count and training tokens must be varied together: increasing student size can eventually raise loss as the available token count falls. Teacher pretraining and inference costs are excluded, isolating student training allocation.
3. Matryoshka Quantization
- https://arxiv.org/abs/2502.06786
- Matryoshka Quantization | Summary
- Matryoshka Quantization, or MatQuant, jointly trains shared integer weights at int8, int4, and int2, representations using 8, 4, and 2 bits. Lower precisions are extracted through rounding, clamping to the target integer range, and rescaling. The shared representation also supplies int3 and int6 models without additional training and supports Mix’n’Match, assigning different precisions to different layers.
- With only feed-forward network weights, the weights in each layer’s feed-forward block, quantized in Gemma-2 2B, Gemma-2 9B, and Mistral 7B, strict int2 MatQuant improves six-task average accuracy over independently trained int2 baselines by 1.04, 3.11, and 3.01 percentage points with OmniQuant, which learns quantization parameters while freezing model weights. With Quantization Aware Training, which updates weights using quantized forward passes, the gains are 4.46, 6.27, and 7.02 points. The OmniQuant int2 baselines receive 20M training tokens versus MatQuant’s 10M.
- The Errata separates strict 2-bit MatQuant from Extra Precison MatQuant, which retains an additional quantization value and uses approximately 2.05 bits per quantized weight on average. On Gemma-2 9B with OmniQuant and co-distillation, the extra-precision variant reaches 70.17 average accuracy; this result has a different representation budget from strict int2 and requires custom handling of extra bits. Intermediate precision does not always match independent training, and the paper benchmarks neither runtime speedups nor end-to-end memory savings.
The nested-bit diagram shows how several precision losses act on shared integer weights, while the histograms show that joint training changes quantized bucket occupancy. The caption’s greater-than-8% int2 claim does not match the corrected strict-int2 comparison in Table 1 and should be interpreted with the extra-precision distinction in the Errata.
Selected mixed-precision configurations fill gaps between homogeneous precision choices, including configurations below 4 effective bits that exceed the int4 slice’s accuracy. Effective bits per FFN parameter represents a weight-storage budget and does not directly measure inference latency.
