SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training | Summary

Contents

Summary
1. Introduction
2. Related Works
3. Preliminaries
4. Evaluation Tasks
- 4.1. The General Points Environment
- 4.2. The V-IRL Environment
5. Results
6. Conclusion, Discussion, and Limitations
Impact Statement
Appendix
Brief Thoughts

This article explains the key points of SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training.

2025-01-28 (arXiv), Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 2025
Chu, Tianzhe, Zhai, Yuexiang, Yang, Jihan, Tong, Shengbang, Xie, Saining, Schuurmans, Dale, Le, Quoc V., Levine, Sergey, Ma, Yi.
HKU, UC Berkeley, Google DeepMind, NYU, University of Alberta
Paper
Github
Project Page

Read this article in Korean

Summary

The paper investigates whether supervised fine-tuning (SFT), training on expert responses, and reinforcement learning (RL), training through rewards for generated responses, help pretrained foundation models handle unfamiliar task rules and visual inputs. Under equal estimated training computation from a shared SFT-initialized checkpoint, RL achieves 45.0% per-step accuracy, the fraction of individual navigation actions judged correct, on an unseen navigation action space, compared with 35.7% at initialization and 2.5% after further SFT.
Experiments use Llama-3.2-Vision-11B on GeneralPoints, a four-card arithmetic task, and V-IRL, a street-view navigation environment, with both text-only and image-and-text inputs. RL uses sequential revision, generating another response after feedback from a verifier, an automated checker that supplies rewards and textual feedback, and improves out-of-distribution (OOD) performance under rules or visual conditions absent from post-training.
Under visual changes with task rules held fixed, RL raises GeneralPoints success from 23.6% to 41.2% and navigation route success from 16.7% to 77.8%, whereas further SFT lowers them to 13.7% and 11.1%. The navigation result exceeds the previously reported 44.0% on the V-IRL mini benchmark by 33.8 percentage points.
The results support RL after sufficient training to follow instructions and produce the required response format in these environments: direct RL without SFT fails, and RL also fails to recover from an extremely overfitted SFT checkpoint. Generalization remains incomplete, with whole-route success under the navigation rule change approximately 1% or lower despite improved per-step accuracy.

1. Introduction

The study separates adaptation to training examples from transfer to new textual rules and visual conditions. It defines memorization as generating near-exact copies of training examples when prompted with information from the training dataset; the experiments primarily assess transfer through controlled task changes rather than directly measuring such reproduction.

Textual generalization changes the instructed interpretation of cards or navigation actions. Visual generalization changes card colors or navigation locations while retaining the task rules.
The pipeline first applies SFT, then trains with multi-turn RL and verifier feedback. The project page is https://tianzhechu.com/SFTvsRL.

Navigation performance under trained and unseen textual action spaces as SFT and RL computation increase.

The paper connects four research areas: post-training, memorization and generalization, inference-time computation, and visual perception. Prior SFT work motivates its role as a teacher of response format, while prior RL work motivates reward-based task learning; the present comparison evaluates both methods with text-only inputs and vision-language inputs, which combine images with text.

Research on inference-time computation, computation spent producing and checking answers during use, motivates repeated verification and revision.
Earlier approaches to visual perception emphasize visual encoders, the model components that represent images, as well as training data and SFT recipes. This study tests whether task-driven RL can also improve recognition.

3. Preliminaries

The method treats response generation as finite-horizon decision making, a sequence with a maximum number of steps: the model is a policy that selects an output from the current input, and training seeks to maximize accumulated rewards. The verifier evaluates each generated outcome and returns a numerical reward and textual feedback; proximal policy optimization (PPO), the policy-update algorithm used here, trains the model.

For text-only inputs, the state contains the input token sequence; for image-and-text inputs, it also contains the image. An action is a generated output sequence.
Sequential revision appends all previous model outputs and verifier feedback to the initial prompt. Subsequent responses can therefore address earlier errors, as illustrated in Figures 2 and 3.

Sequential revision of a four-card arithmetic answer using verifier feedback and rewards.

Prompt construction with task rules, model outputs, and accumulated verifier feedback.

4. Evaluation Tasks

The evaluation uses two environments that each support textual rule changes and visual changes. GeneralPoints tests arithmetic reasoning, while V-IRL tests spatial reasoning and recognition during navigation; each has a text-only variant, denoted -L, and an image-and-text variant, denoted -VL.

4.1. The General Points Environment

GeneralPoints requires an equation equal to 24 using each of four card values exactly once. GP-L supplies card descriptions as text, while GP-VL supplies an image and requires the model to recognize the cards before solving the arithmetic problem.

The rule-change experiment trains with ‘J’, ‘Q’, and ‘K’ all valued at 10, then evaluates them as 11, 12, and 13, respectively. The applicable rule appears in the prompt, and evaluation requires at least one face card.
The visual-change experiment trains on black suits, ♠ and ♣, and evaluates on red suits, ♥ and ♦. This tests whether card recognition transfers across color.

4.2. The V-IRL Environment

V-IRL requires following spatial instructions to a destination while identifying relevant landmarks. V-IRL-L supplies textual observations, and V-IRL-VL supplies street-view images; Figure 4 illustrates how landmark observations determine when the agent turns or stops.

The rule-change experiment replaces absolute headings, such as ‘north’ and ‘northeast’, with relative turning actions: ‘left’, ‘right’, ‘slightly left’, and ‘slightly right’. These correspond to turns of 90 degrees or 45 degrees; moving forward and stopping remain available.
The visual-change experiment trains on New York City routes and evaluates on a worldwide route benchmark, testing transfer of landmark recognition and navigation. The benchmark includes New York routes that do not overlap with training.

A V-IRL route combining street-view landmarks, navigation instructions, turns, and a destination.

5. Results

The main experiments use Llama-3.2-Vision-11B and initialize both comparison branches from an SFT-trained checkpoint before separately increasing SFT or RL computation. All model components are trainable in the main experiments, using one shared learning rate across components within each experiment.

The results examine rule transfer, visual transfer, recognition accuracy, the need for SFT initialization, and the number of verification iterations.
Compute comparisons estimate floating-point operations, reported in billions of operations (GFLOPs), and include the additional generation cost required to collect RL training responses.

5.1. Generalization across Rules

Across all four task variants, RL improves performance on unseen rules, while further SFT reduces it. Figures 5 and 6 compare training dynamics and endpoints from a shared initialization, with equal estimated computation for the endpoint comparisons.

GeneralPoints episode success, obtaining at least one correct answer within the allowed verification attempts, changes from 11.5% to 15.0% with RL and to 3.4% with SFT in GP-L; in GP-VL, it changes from 11.2% to 14.2% with RL and to 5.6% with SFT.
V-IRL per-step accuracy changes from 80.8% to 91.8% with RL and to 1.3% with SFT in V-IRL-L; in V-IRL-VL, it changes from 35.7% to 45.0% with RL and to 2.5% with SFT.
These navigation values measure individual actions. Figure 18 shows whole-route OOD success under the rule change at approximately 1% or lower, so the action-level gains do not establish reliable route completion.

Training-compute trends for in-distribution and unseen-rule performance across four task variants.

Unseen-rule performance of RL, shared initialization, and continued SFT under equal estimated computation.

5.2. Generalization in Visual Out-of-Distribution Tasks

With task rules held fixed, RL also improves transfer to unfamiliar visual inputs. Figure 7 reports GeneralPoints success of 41.2% after RL versus 23.6% at initialization and 13.7% after SFT, and V-IRL route success of 77.8% after RL versus 16.7% at initialization and 11.1% after SFT.

GeneralPoints changes from black to red suits. V-IRL changes from New York City training routes to the original worldwide navigation mini benchmark.
The 77.8% result exceeds the previously reported 44.0% by 33.8 percentage points. The earlier system used a two-stage vision-language/language-model pipeline and tailored prompting, so this is a benchmark comparison between different systems.

5.3. RL Improves Visual Capabilities

The GP-VL analysis measures recognition of the four input cards alongside equation-solving success under both rule and visual changes. Figure 8 shows that increasing RL computation improves recognition accuracy and episode success together, while increasing SFT computation degrades both in these experiments.

The relationship supports visual recognition as a contributor to task performance, but the analysis does not isolate recognition improvements as the sole cause of better reasoning.
Additional learning-rate and component-freezing experiments, which keep selected model components fixed during training, investigate the poor GP-VL SFT performance in Figures 16 and 17.

Card-recognition accuracy versus GP-VL episode success under rule and visual variants.

5.4. The Role of SFT for RL Training

Direct RL on the base model without SFT initialization fails to improve GP-L performance in every reported trial. Figure 9 shows success approaching zero, and Figure 20 illustrates a response that attempts to provide Python code instead of the required structured answer.

The authors attribute this failure to poor instruction following, which prevents reliable extraction of task information and rewards.
The finding is conditional on the tested backbone and setup. The paper explicitly distinguishes it from results on other models where RL without SFT succeeds.

GP-L RL training without SFT initialization at learning rates 1e-6, 2e-6, and 5e-7.

5.5. Role of Verification Iterations

GP-L experiments vary the maximum number of verification iterations over {1, 3, 5, 10}. Under the same computational budget, OOD performance increases by 0.48, 2.15, 2.99, and 5.99 percentage points, respectively, supporting a benefit from additional opportunities to revise responses.

Figure 10 compares in-distribution growth, improvement on the trained rule, with OOD growth. More iterations produce stronger OOD gains even though the largest in-distribution gain occurs with one iteration.

In-distribution and OOD performance growth with 1, 3, 5, or 10 verification iterations.

6. Conclusion, Discussion, and Limitations

The paper concludes that its RL pipeline transfers better than continued SFT across the tested arithmetic and navigation changes, while identifying unresolved problems in both methods. SFT fails to approach RL’s in-distribution GP-VL performance despite additional experiments, and RL remains sensitive to the quality of its starting checkpoint.

The authors hypothesize that SFT overemphasizes more frequent reasoning tokens and neglects tokens used to report visual recognition. This explanation remains a hypothesis.
RL does not recover rule-transfer performance from an extremely overfitted navigation checkpoint whose initial OOD per-step accuracy is below 1%, as shown in Figure 19. Figure 21 illustrates the model producing an absolute heading when the prompt requires a relative direction.
Together with the failed runs without SFT, these experiments identify failures at both insufficiently instruction-trained and severely overfitted initializations.

Whole-route performance under the navigation rule change remains approximately 1% or lower even though individual-action accuracy improves elsewhere in the paper. Requiring a sequence of correct actions makes route completion a substantially harder test of transfer.

Starting from a severely overfitted checkpoint, both tested RL learning rates remain near zero OOD per-step accuracy without sustained recovery. The result limits the claim that RL can repair damage caused by excessive SFT.

Impact Statement

The Impact Statement describes GeneralPoints and V-IRL as controlled research environments, with navigation serving as a simulated proxy for real-world tasks. The authors state that no deployment or interaction with actual real-world systems or data was involved and identify no foreseeable societal or ethical implications from these experiments.

Appendix

A. Details on the General Points Environment, A.1. Data, A.2. Detailed Examples on the Transition Dynamics, and A.3. Additional Eetails on the Environmental Design specify sampling, revision examples, configuration options, and rewards. Four cards are sampled from a standard 52-card deck, with an expert solver ensuring a solution equal to 24; rule-transfer evaluation requires at least one face card, while configurable options include the target, face-card rule, sampling, and suit colors. Figures 11 and 12 illustrate image-and-text and text-only revision, respectively. Episodes end upon success or after 5 verification steps: rewards are 5 for a correct legal equation, −1 for an incorrect legal equation or exhausting verification steps, −2 for using unavailable numbers, and −3 for other illegal equations, with an additional −1.5 penalty for incorrect card recognition in GP-VL. These specified rewards differ from the illustrative values in Figure 2.
B. Details on the V-IRL Environment, B.1. Data, B.2. Detailed Examples on the Transition Dynamics, and B.3. Additional Details on the Environmental Design describe 1000 unique New York City training routes, navigation examples, and the simplified environment. Visual-transfer evaluation uses 18 routes across nine cities, two per city, with its New York routes excluded from training; routes include street views, instructions, expert observations, and expert trajectories, the sequences of correct navigation actions. Figures 13 and 14 illustrate image-and-text and text-only transitions. The implementation removes the separate visual detector and online queries, supports absolute or relative turning actions plus forward() and stop(), and limits straight-road length; rewards are 1 for a correct action, −1 for a wrong action or exceeding the maximum verification step of 2, and −1.5 for failed landmark detection.
C. Experimental Setup, C.1. Data, C.2. Training Pipeline, and C.3. Evaluation Metric describe expert prompt-response pairs, SFT initialization, compute accounting, and evaluation. Main-body SFT uses optimal single-turn examples without revision; a GP-L ablation adds erroneous trajectories and verifier messages but still shows deteriorating OOD performance. Training uses 8 H800 graphics processing units with 80GB memory each; GeneralPoints success requires at least one correct answer during verification, while navigation route success requires correct actions at every movable point, and per-step accuracy counts intermediate verification attempts independently. Compute estimates include initialization, optimization, and RL response generation. Curves use a Savitzky–Golay filter, a local polynomial smoothing method, of polynomial order 3, and error bars approximate binomial standard errors, uncertainty estimates for the measured success proportions.
D. Additional Experimental Results, D.1. Ablation Studies on GP-VL, D.2. More results on V-IRL-VL, and D.3. Failure Cases provide hyperparameter checks and failure evidence. SFT experiments vary learning rates and freeze either the vision encoder or both the encoder and adapter, the component linking visual representations to the language model, without a sustained increase beyond 30% in-distribution success; RL improves at both tested learning rates, 1 × 10⁻⁶ and 2 × 10⁻⁶. Figure 18 documents the gap between per-step and whole-route rule-transfer performance, while Figures 19–21 show failure to recover from overfitting and failure to follow the required output format without SFT.

Brief Thoughts

The strongest evidence is the repeated divergence between continued SFT and RL from shared checkpoints across two tasks, two input formats, and separate rule and visual changes. Equal estimated compute strengthens the comparison, and the GP-L experiment with erroneous trajectories and verifier messages shows that adding this form of trajectory diversity does not remove SFT’s transfer failure in that setting.

The title should be read within the tested conditions: the main evidence concerns transfer performance on one backbone rather than direct measurement of near-exact training-example reproduction. Navigation rule-transfer success remains very low at the whole-route level. The printed card prompts in Figures 12 and 20 also contain conflicting face-card instructions; the PDF alone does not establish whether these inconsistencies occur in the actual evaluation prompts.