Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model | Summary

This article explains the key points of Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model.

• 2025-10-21 (arXiv)
• Ling Team, Shen, Anqi, Li, Baihui, Hu, Bin, Jing, Bin, Chen, Cai, Huang, Chao, Zhang, Chao, Yang, Chaokun, Lin, Cheng, et al.
• Inclusion AI
• Paper
• Github

Read this article in Korean

Summary

• Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model addresses unstable updates and slow response generation when reinforcement learning, training from rewarded model outputs, is applied to a trillion-parameter reasoning model. Ring-1T scores 93.40 on AIME 2025, a competition mathematics benchmark, versus 92.30 for Qwen3-235B-A22B-Thinking-2507 and 94.60 for GPT-5-Thinking (High) in Table 1’s Avg@64 comparison, which averages 64 sampled responses and uses the higher of reproduced and official baseline scores when both are available.
• Ring-1T has 1 trillion total parameters and approximately 50 billion activated parameters per token through a Mixture-of-Experts (MoE) architecture, which routes each token through selected expert components. Training combines supervised learning from extended reasoning sequences, reasoning reinforcement learning with rewards from verifiable outcomes, and general reinforcement learning informed by human feedback.
• IcePop suppresses token updates when training and inference engines assign excessively different probabilities; C3PO++ partitions rollouts, generated response trajectories, across policy updates under a token budget. Against a baseline without this partition mechanism, the paper reports approximately 2.5 times rollout speedup and about 1.5 times end-to-end speedup per step, with similar measured benchmark performance.
• Ring-1T reaches a CodeForces competitive-programming rating of 2088, scores 55.94 on ARC-AGI-1, a visual pattern reasoning benchmark, and solves four International Mathematical Olympiad (IMO) 2025 problems on its first attempt within AWorld, a multi-agent framework. Its weaker software-editing, knowledge, instruction-following, and multi-turn scores indicate an uneven capability profile.

1 Introduction

The introduction identifies training instability and computational cost as obstacles to trillion-parameter reasoning models. Ring-1T starts from Ling-1T-base using the Ling 2.0 architecture, and the report presents three connected contributions: IcePop for training-inference probability mismatch, C3PO++ for long-response scheduling, and ASystem for distributed training infrastructure.

Ring-1T exceeds the other listed open-weights models on selected mathematics, ARC-AGI-1, preference-alignment, and healthcare tasks, while GPT-5-Thinking (High) scores higher on those panels. Ring-1T-Preview has a CodeForces rating of 2092 versus Ring-1T’s 2088, so this overview does not support a universal highest-rating claim.

2.1 Training Pipeline

The Approach uses three stages: Long Chain-of-Thought Supervised Fine-Tuning (Long-CoT SFT), supervised training on extended step-by-step reasoning trajectories; reasoning reinforcement learning; and general reinforcement learning. The report focuses primarily on the reinforcement learning components.

• Reasoning training uses Reinforcement Learning from Verifiable Rewards (RLVR), with rewards supplied by domain-specific checks for mathematics, code, science, and logic.
• General training uses Reinforcement Learning from Human Feedback (RLHF), training informed by human preferences, to improve instruction following, writing, safety, and usability.

The diagram separates learning extended reasoning patterns from reward-based reasoning improvement and later preference alignment. Different data corpora feed each stage, showing that the final model results from the combined training pipeline.

2.2 Long-CoT SFT

Long-CoT SFT queries come from public repositories, expert authors, and large language models (LLMs), language-generation models used here to synthesize training examples. Cleaning applies exact deduplication, harmful-content filtering, benchmark decontamination, removal of training examples overlapping evaluation problems through hashing and exact string matching, and low-quality filtering for control codes and extraneous Unicode characters.

• The main text reports Mathematics 46%, STEM, science, technology, engineering, and mathematics, 26%, Code 20%, and Others 8%; Appendix D presents different detailed proportions.
• Sequences are packed to 64k length and trained for 3 epochs with learning rate 2 × 10⁻⁴, cosine decay, a schedule that gradually lowers the learning rate, 30 warmup steps, and weight decay 0.1, a parameter regularization setting.

2.3 Large-Scale Reinforcement Learning

Large-scale reinforcement learning combines an algorithmic response to unstable updates with infrastructure for generation and training. IcePop controls engine mismatch, C3PO++ manages long trajectories, and ASystem supports their execution at Ring-1T’s scale.

2.3.1 RL Data

RL Data spans mathematics, code, science, logic, and general tasks. Mathematics problems require verifiable solutions; code problems receive synthesized and validated test cases; and science questions are converted from multiple-choice to open-ended formats, with molecular images converted into structured text.

• Logic data includes visual pattern induction, inferring transformations from examples, Sudoku grid puzzles, maze pathfinding, the 24 Game for arithmetic reasoning, and Knights and Knaves for propositional logic; generation and post-processing checks enforce solvability and non-triviality.
• General data combines public datasets, human preference records, real user interactions, and questions from Zhihu and StackOverflow.
• Parsing, reformulation, deduplication, model-based scoring, and rule-based metrics support curation; sample metadata enables dynamic sampling and cross-domain blending.

2.3.2 IcePop: Discard All Noisy Gradient Updates

IcePop addresses probability discrepancies between separate training and inference engines, which can be amplified by dynamic expert routing and long reasoning sequences. It modifies GRPO, the group-based policy optimization baseline used in the paper, by weighting token updates with a training-to-inference probability ratio and discarding updates outside a specified interval.

• For calibration ratio k = π_train(a; θ_old) / π_infer(a; θ_old), where the two probabilities are computed by the training and inference engines at the old parameters, the mask is M(k) = k when k ∈ [α, β], and 0 otherwise.
• The objective retains a clipped policy-update term, which limits the probability ratio used in the update, and a reference-policy Kullback–Leibler (KL) divergence term, a measure of distribution difference; the reported reinforcement learning recipe sets the latter’s coefficient to 0.0.
• The compounding-discrepancy theorem is conditional: Appendix B requires smoothness, bias alignment, bounded on-policy drift, and a local gradient bound, with multiplicative growth established only above a discrepancy threshold.

C3PO++ schedules trajectory generation across updates, while IcePop controls which token updates contribute gradients according to engine probability ratios. The shared policy and separate engines show the distinct roles of generation scheduling and update calibration.

2.3.3 C3PO++: Dynamically Partition Rollouts with Budget

C3PO++ maintains an inference pool for parallel response generation and a training pool for completed trajectories. Once completed trajectories contribute enough tokens to reach budget Φ, the model updates; unfinished trajectories remain buffered and resume using the updated policy.

• Algorithm 1 increments the token counter by the full length of each completed trajectory, so the operational budget concerns completed training trajectories.
• A retention period counts the partitions experienced by an unfinished trajectory; trajectories exceeding threshold σ are removed, and new prompts refill available inference capacity.
• Completed responses can contain segments generated by earlier policy versions, so the method reuses trajectories across parameter updates.

Hatched buffered segments persist across training steps and are completed by newer policy versions. Training proceeds with completed responses while unfinished responses remain available for continuation, producing trajectories with segments from multiple policy versions.

2.3.4 Training Recipe

All policy optimization uses ASystem and AdamW, the parameter-update optimizer, with β₁ = 0.9, β₂ = 0.999, weight decay 0.01, and fixed MoE router bias, a setting in the component that selects experts. Reasoning reinforcement learning uses IcePop with α = 0.5 and β = 5 together with C3PO++, learning rate 2 × 10⁻⁶, KL coefficient 0.0, and sampling temperature 1.0, the setting controlling generation randomness.

• Each reasoning step uses 480 unique prompts with 8 rollouts per prompt and a maximum length of 65,536 tokens.
• General reinforcement learning uses GRPO, learning rate 3 × 10⁻⁶, KL coefficient 0.0, and temperature 1.0; each step uses 80 unique questions with 8 outputs each and maximum length 32,768 tokens.

2.4 Experiments and Analysis

Experiments and Analysis evaluates IcePop through benchmark learning curves and training diagnostics, and C3PO++ through timing, reward, and benchmark comparisons. These experiments address update stability and rollout efficiency separately; they do not isolate every interaction among the full pipeline’s components.

2.4.1 IcePop

The preliminary IcePop comparison uses Ring-mini-2.0, an MoE model with 16.8B total and 0.75B activated parameters, with the same training dataset across settings. It compares IcePop with α = 0.5 and β = 5, TIS, an importance-sampling correction that reweights updates to account for engine mismatch, using its recommended setting, and vanilla GRPO without the KL term.

• The AIME25 Avg@64 curve rises from approximately 63 to approximately 77 with IcePop, a gain of about 14 percentage points; the paper describes the final gap over TIS as a relative 6%.
• On Ring-1T, the IcePop and GRPO curves branch around step 300; IcePop restrains the subsequent increase in gradient norm, the magnitude of the parameter-update gradient, and extreme training-inference log-probability discrepancy.
• The plotted diagnostics support stabilization in these runs while retaining nonzero numerical disagreement between engines.

After the curves branch around step 300, IcePop restrains the sharp late increase in gradient norm and maximum log-probability discrepancy observed under GRPO. The average discrepancy remains nonzero, consistent with the paper’s acknowledged residual engine mismatch.

2.4.2 C3PO++

C3PO++ is compared with a baseline that omits budget-controlled rollout partitioning. The text reports approximately 2.5 times faster rollouts and about 1.5 times faster end-to-end execution per step, with closely tracking reward curves.

• Baseline versus C3PO++ scores are 92.29 versus 92.29 on AIME25, 2084 versus 2085 on CodeForces, and 53.62 versus 53.25 on ARC-AGI-1.
• These results show similar performance at reduced step time for the tested configuration; the paper does not report confidence intervals for these differences.

Both timing panels place C3PO++ below the baseline across most plotted steps. The text reports approximately 2.5 times rollout speedup and about 1.5 times end-to-end speedup; the plotted end-to-end levels suggest a larger ratio, and the T axis supplies no absolute duration or explanation of that difference.

Reward curves overlap substantially, and the benchmark bars show identical AIME25 scores, a one-point CodeForces rating increase, and a 0.37-point ARC-AGI-1 decrease with C3PO++. These measurements show similar performance in the tested comparison alongside the timing improvement.

2.5 Large-Scale RL Infrastructure: ASystem

ASystem combines Hybrid Runtime for training-inference execution, AMem for accelerator memory management, AState for weight synchronization, and ASandbox for isolated verification. Its SingleController + SPMD architecture combines centralized orchestration with Single Program, Multiple Data execution across workers, separates control flow from data flow, and supports interchangeable training, inference, and reward backends.

• The overview reports synchronization of trillion-parameter models in under 10 seconds and describes automatic recovery from slow training and hangs.
• The infrastructure description provides mechanisms and selected performance claims but no detailed hardware configuration or component-by-component scaling benchmark.

The architecture assigns algorithm orchestration to AReaL, unified execution to Hybrid Runtime, memory management to AMem, weight exchange to AState, and isolated verification to ASandbox. This division explains the proposed response to distributed training bottlenecks; the schematic itself contains no scaling measurements.

2.5.1 Hybrid Runtime: A Unified Training-Inference Execution Environment

Hybrid Runtime integrates resource scheduling, parallel execution, training, and inference. AState transfers only necessary weight shards, distributed portions of model parameters, between peers and updates inference weights in place, selecting communication transport according to data size and hardware topology.

• This subsection reports sub-second parameter updates, while the preceding overview gives under 10 seconds for trillion-parameter synchronization; the report does not specify whether these refer to the same workload or measurement.
• AMem releases and resumes training state through Memory Switching, aggregates bandwidth through Distributed Multi-path Transfer, and dynamically allocates memory through Unified Memory Pooling across accelerators and nodes.

2.5.2 ASandbox: An On-Demand Serverless Sandbox Engine

ASandbox executes verification tasks in isolated environments, including code execution and terminal simulation, and integrates with Kubernetes, the cluster-management system. The paper reports secure-container isolation using runsc and kata, automatic node-failure detection and isolation, 100ms startup, and a throughput specification of 5,000 QPS/200ms, where QPS denotes queries per second; it does not explain the latter specification’s measurement conditions.

2.5.3 AReaL: A High-Performance RL Algorithm Framework

AReaL is ASystem’s open-source reinforcement learning framework with single-controller and SPMD interfaces. Its asynchronous pipeline overlaps trajectory generation, reward computation, and training; packing sequences together and distributing data across workers reduce padding and rebalancing, while automated detection, retry, and recovery address execution failures.

3 Evaluation

Evaluation compares Ring-1T with open-weights models, whose parameters are available, and proprietary reasoning models across mathematics, coding, logical reasoning, and general tasks. Table 1 extends the comparison beyond the reasoning tasks emphasized during training.

3.1 Benchmarks

Benchmarks cover eight domains: knowledge, coding, mathematics, reasoning, alignment with human preferences, healthcare, multi-turn interaction, and agent capabilities. Comparators in Table 1 are DeepSeek-V3.1-Terminus-Thinking, Qwen3-235B-A22B-Thinking-2507, Gemini-2.5-Pro, and GPT-5-Thinking (High).

• Coding evaluation includes LiveCodeBench-v6 (2408-2505), a code-generation benchmark, CodeForces, and Aider, a software-editing benchmark; CodeForces uses problems from 14 Div. 2 contests with expert-designed test cases and a maximum attainable rating of 2209.
• General evaluation includes ArenaHard v2 for preference alignment, HealthBench for healthcare, MultiChallenge for multi-turn interaction, and BFCL v3, the Berkeley function-calling benchmark.

3.2 Evaluation Settings

AIME 2025, LiveCodeBench-v6, ARC-AGI-1, CodeForces, HMMT 2025, a competition mathematics benchmark, ZebraLogic, a logic benchmark, HLE, Humanity’s Last Exam, and BFCL v3 use a 128K context window, the token capacity available during evaluation; models with insufficient native context are extended using YaRN, a context-window extension method. Aider, CNMO 2024, another competition mathematics benchmark, and BBEH, Big-Bench Extra Hard, use 64K, and remaining benchmarks use 32K.

• Open-weights baselines use official hyperparameters and proprietary models use vendor-recommended settings.
• When official baseline results exist, the paper selects the higher score between its reproduction and the official release; reported comparisons therefore combine reproduced and externally reported results.
• Table 1 specifies Avg@64 for AIME 2025, Avg@16 for HMMT25, and Avg@4 for LiveCodeBench-v6, averages over the indicated numbers of samples, while Figure 1 labels these plots pass@1, single-response success.

3.3 Results

Table 1 shows Ring-1T leading the listed open-weights models on AIME 2025 at 93.40, HMMT25 at 86.72, LiveCodeBench-v6 at 78.30, ARC-AGI-1 at 55.94, ArenaHard v2 win-rate at 81.59, and HealthBench at 57.93. GPT-5-Thinking (High) scores higher on each of these benchmarks, while Ring-1T leads the table’s CodeForces comparison with rating 2088 and percentile 97.85.

• ARC-AGI-1 gains are 15.32 percentage points over DeepSeek-V3.1-Terminus-Thinking and 7.82 over Qwen3-235B-A22B-Thinking-2507.
• Within AWorld, Ring-1T solves IMO Problems 1, 3, 4, and 5 on the first attempt through natural language reasoning without code generation or external symbolic solvers, corresponding to the paper’s silver medal-level result. A nearly complete Problem 2 proof appears on the third attempt; Problem 6 produces 4048 instead of the correct 2112.
• The broad table also shows weaker results: Aider 78.57 versus DeepSeek’s 92.86, MMLU-Pro, a multi-task knowledge benchmark, 80.54 versus 85.00, IFEval, an instruction-following benchmark, 85.21 versus 89.09, and MultiChallenge 50.92 versus Qwen’s 52.75.
• Figure 1 includes Ring-1T-Preview with CodeForces rating 2092, so the highest-rating claim for Ring-1T applies to the models listed in Table 1.

The full comparison shows selected reasoning strengths alongside weaker software-editing, knowledge, instruction-following, and multi-turn results. Ring-1T activates 50B parameters per token versus 37B for DeepSeek and 22B for Qwen, and dagger-marked entries come from official documentation, limiting causal attribution under matched computation.

4 Conclusion

The conclusion attributes successful Ring-1T training to the combined use of IcePop, C3PO++, and ASystem. The reported experiments demonstrate feasibility and strong selected benchmark results for an open-weights trillion-parameter reasoning model, while leaving the contribution of parameter scale confounded with data, algorithms, and infrastructure.

5 Limitations & Future Work

The authors identify costly long-response inference, residual numerical disagreement between training and inference operators, and under-optimized agentic skills such as tool use. They also report identity confusion and linguistic code-switching, switching languages within responses, which they attribute to data impurity and insufficient regularization.

• The current attention architecture remains costly during extensive reasoning; alternative attention mechanisms and specialized agent training are proposed future work.
• IcePop mitigates mismatch but does not eliminate its underlying numerical causes, consistent with the nonzero discrepancy in the training diagnostics.

6 Contributors

The title page identifies Ling Team, Inclusion AI, and Section 6 lists contributors alphabetically by first name. Jun Zhou, Zhiqiang Zhang, and Zujie Wen are marked as corresponding authors.

Appendix

• A Related Work contains A.1 Stable Reinforcement Learning, A.2 Efficient Reinforcement Learning, and A.3 Reinforcement Learning Infrastructure. It distinguishes IcePop’s token masking from sequence-level policy optimization, which computes an update ratio for a whole response, and TIS correction; compares C3PO++ with truncated and interruptible generation; and motivates ASystem through memory, synchronization, and reproducibility bottlenecks.
• B Theoretical Analysis for IcePop defines δ_t as the KL discrepancy between inference and training distributions and proves δ_(t+1) ≥ (1 + ημ)δ_t − κμ under assumptions A1–A4: smooth discrepancy, update bias aligned with discrepancy growth, bounded on-policy drift, the contribution from updates sampled from the training distribution, and bounded local gradients. The step size must satisfy 0 < μ ≤ μ̄, with the assumptions holding throughout the trajectory; for δ_t ≥ 2κ/η, the bound becomes δ_(t+1) ≥ (1 + ημ/2)δ_t. This establishes conditional mismatch amplification, not a convergence guarantee for IcePop.
• C Preliminary Analysis for IcePop on Ring-mini-2.0 reports GRPO reward collapse after 180–200 steps, whereas IcePop and TIS remain stable through 600 steps. IcePop shows smaller extreme engine discrepancies than TIS, and the authors interpret lower token log probabilities as evidence of greater exploration. The text describes approximately 1–2‰ of tokens as masked, with a higher proportion of high-entropy tokens, tokens with greater predictive uncertainty, although Figure 12 rises above that range late in training; sensitivity tests find [0.5, 2.0] unstable, while [0.5, 5.0] and [0.4, 5.0] remain stable in the reported runs.
• D Training Data Analysis shows detailed supervised-data proportions of Math 43%, Code 18%, Physics 13%, and Biology 4%, differing from the main text’s Mathematics 46% and Code 20% without explanation. The reasoning-data chart reports domain shares of Math 42.2%, Code 18.6%, Science 17.8%, and Logic 21.3%, with Very Hard labels of 35%, 38%, 29%, and 30%, respectively. The displayed difficulty percentages within each domain do not sum to 100%, and the appendix does not explain the shortfalls.
• E IMO Case reproduces Ring-1T’s answer to Problem 1: for n ≥ 3, cover all positive integer points (a, b) with a + b ≤ n + 1 using n distinct lines, exactly k of which are sunny, meaning parallel to neither coordinate axis nor x + y = 0. Inside the reproduced answer’s Summary, a. Verdict states k = 0, 1, 3, and b. Method Sketch outlines counting bounds and explicit constructions. These lowercase labels belong to the reproduced answer under E IMO Case and are separate from Appendix A and Appendix B.
• Detailed Solution begins with Step 1: Understanding the Point Set, defining the triangular grid with n(n + 1)/2 points, and Step 2: Maximum Coverage by Non-Sunny Lines, asserting M(m, n) = m(2n − m + 1)/2 and U(k) = k(k + 1)/2. Step 3: Impossibility for Even k ≥ 2 and Step 4: Impossibility for Odd k ≥ 5 use claimed coverage and collinearity bounds to exclude those cases. These arguments are reproduced model output; the appendix provides no separate justification for assuming that the uncovered points have the triangular structure required by those bounds.
• Step 5: Possibility for k = 0 uses x + y = 2, …, n + 1, and Step 6: Possibility for k = 1 uses n − 1 horizontal lines plus a sunny line through (1, n). Step 7: Possibility for k = 3 leaves a six-point triangular subgrid covered by lines with slopes 1, −1/2, and −2. Step 8: Completeness of Characterization combines the stated exclusions and constructions, and Final Answer repeats k = 0, 1, 3.

Brief Thoughts

The clearest engineering evidence is the restrained mismatch diagnostics after IcePop and faster C3PO++ steps with similar measured task scores. The broader model comparison supports selected reasoning strengths, but differing activated parameter counts and mixed reproduced/official baseline scores limit attribution to the proposed methods. Missing uncertainty estimates, unexplained data-proportion differences, and unspecified synchronization measurement conditions constrain reproducibility.

The release provides a model and implementation reference: https://huggingface.co/inclusionAI/Ring-1T and https://github.com/inclusionAI/Ring-V2. Its benchmark profile supports evaluating competition reasoning, practical software editing, and agent workflows separately when assessing suitability for an application.