DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization | Summary

Contents

Summary
1 Introduction
2 Related Work
- 2.1 Preference Optimization for LLMs
- 2.2 Dual Learning
3 Dual Learning-based Preference Optimization
4 Experiment
5 Conclusion
6 Limitations
Appendix
Brief Thoughts

This article explains the key points of DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization.

2025-08-20 (arXiv)
She, Shuaijie, Bao, Yu, Lu, Yu, Xu, Lu, Li, Tao, Zhu, Wenhao, Huang, Shujian, Cheng, Shanbo, Lu, Lu, Wang, Yuxuan.
ByteDance Seed, Nanjing University
Paper

Read this article in Korean

Summary

Improving a large language model (LLM), a model that generates text for tasks such as translation and mathematical reasoning, usually requires externally supplied feedback or answer labels. Compared with the original Qwen3-4B under the paper’s evaluation protocol, DuPO raises average accuracy on three mathematics contest benchmarks from 77.2% to 83.6%, a 6.4 percentage-point gain.
DuPO, Dual Learning-based Preference Optimization, scores an output through a complementary task that reconstructs a hidden input component using the output and the remaining input. It generalizes dual learning, supervision through paired forward and reverse tasks, by replacing full input reconstruction with this conditional reconstruction.
Seed-X-7B-Instruct gains 2.13 COMET points, a neural translation-evaluation score, across 756 translation directions; training-free reranking, selecting among generated candidates by a reconstruction score, raises Qwen3-4B’s average accuracy on two mathematics benchmarks from 68.4% to 77.7% with additional reverse-task sampling. The experiments cover translation and mathematical reasoning, while component selection adds computation and some mathematical results differ between the main table and the ablation.

1 Introduction

The introduction identifies external supervision as a bottleneck in model improvement. Reinforcement Learning from Human Feedback (RLHF), training with human preference signals, requires costly and potentially inconsistent annotations; Reinforcement Learning with Verifiable Rewards (RLVR), training with rewards checked against definitive answers, still requires those answers.

Translation can have multiple valid outputs, so a single reference need not represent every high-quality response.
Traditional reverse-task supervision faces two obstacles: an output may not identify its input uniquely, and the model may perform the forward and reverse tasks with unequal competence.
DuPO supplies known input components to the reverse task and reconstructs only selected unknown components, using one model for both directions.

The related work connects preference optimization, improving output generation using feedback scores, with dual learning. It compares supervision from human preferences, model judgments, externally verifiable answers, and consistency between paired tasks.

2.1 Preference Optimization for LLMs

The paper contrasts human preference rewards, LLM-as-a-Judge, a language model evaluating an output, and verifiable-answer rewards. Human annotation introduces cost and disagreement, model judgments depend on judge competence and can reflect presentation-order or linguistic-style biases, and answer-based rewards require definitive reference outcomes.

DuPO derives feedback from reconstruction consistency rather than a separate preference annotation or a supplied final answer.
Its implementation still uses task-specific reconstruction checks.

2.2 Dual Learning

Dual learning uses paired tasks to supervise each other, as in translation followed by back-translation, translating an output back into its source language. The paper reviews extensions across modalities, knowledge reasoning, and reinforcement learning, as well as Trans-Zero and DualReflect for language-model translation.

Cycle consistency means that applying the reverse task recovers the original input within an acceptable error.
DuPO replaces full reconstruction with conditional reconstruction of a selected input component, accommodating mathematical problems whose answers cannot uniquely identify the original question.

3 Dual Learning-based Preference Optimization

DuPO converts relationships between a task and its complementary counterpart into self-supervised rewards, feedback derived from reconstruction of the input. The method formalizes reconstruction consistency, identifies failures of full reversal, and optimizes output generation using partial-reconstruction rewards.

3.1 Task Duality

Definition 1 specifies a primal task, the original forward mapping, and a dual task, its reverse mapping, with a domain-specific reconstruction distance bounded by a nonnegative tolerance. Equation (1) assigns greater reward to outputs whose reverse reconstruction is closer to the original input: r(y) ∝ exp(−λ · d(x, T_d(y))).

The positive parameter λ controls reward sensitivity to reconstruction error.
Useful feedback requires sufficient information in the output and reliable reverse-task reconstruction.

3.2 Challenges in Dual Learning-based Optimization

Full reversal fails when multiple inputs produce the same output: the answer 8 cannot distinguish a ball-counting question from other questions with that answer. Even when reversal is possible, bidirectional competence asymmetry, unequal forward and reverse performance, can make reconstruction error penalize a correct output.

The translation example shows how reconstructing “fast” instead of “quick” can create a mismatch despite a correct German translation.
Figure 1 separates reconstruction ambiguity from reverse-task failure and illustrates reconstruction of a smaller component as the proposed remedy.

Challenges in dual learning and solutions via relaxed duality constraints.

3.3 Generalized Duality Framework via Complementary Task

Definition 2 decomposes the input into known and unknown components and defines a complementary dual task that reconstructs the unknown component from the output and known context. The pair must satisfy a partial-reconstruction error bound; Equation (2) defines its reward as r(y) ∝ exp(−λ · d(x_u, T_cd(y, x_k))).

In the two-sum example, the forward task computes C = A + B; given C and A, the complementary task reconstructs B′ = C − A and checks whether B′ matches B.
Known context constrains reconstruction, while selecting a feasible unknown component reduces reverse-task difficulty.
The paper proposes applications to code generation and dialogue, but evaluates mathematics and translation.

3.4 Preference Optimization

Equation (3) maximizes expected reconstruction reward under the model’s output policy, its conditional distribution over generated outputs: J(θ) = E_{y∼π_θ(y|x)}[r(y)]. The experiments use Group Relative Policy Optimization (GRPO), a reinforcement-learning optimizer, to train the model with these rewards.

For translation, the paper describes BLEU, an automatic translation score, as a reconstruction measure; for mathematics, equality with the hidden numerical value yields a binary reward.
The framework permits other reinforcement-learning optimizers, but the reported experiments use GRPO.

4 Experiment

The experiments evaluate multilingual translation and mathematical reasoning. They compare trained models, additional model architectures, direct training of a base model, inference-time reranking, and an ablation, a controlled removal of component selection to assess its contribution.

4.1 Experiment Setup

Translation training uses Seed-X-7B-Instruct with 1,000 prompts per language across 28 languages and 7,000 parallel entries from the Flores-200 development set, a multilingual translation dataset. Evaluation samples 50 examples for each of 756 translation directions from the Flores test set, totaling 37,800 examples, and reports BLEU, COMET, and BLEURT, a learned text-generation evaluation metric.

Human evaluation on Seed-X-Challenge covers 14 directions from Chinese or English to seven languages, with experts scoring accuracy, fluency, and idiomaticity on a 0–4 scale.
Mathematics experiments use DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, Qwen3-4B, and OpenReasoning-Nemotron-7B on AMC23, AIME24, and AIME25, mathematics contest benchmarks.
For each mathematics problem, evaluation samples 32 responses at temperature 0.8, the sampling setting controlling output randomness, with a maximum reasoning budget of 32,000 tokens and reports Avg@32, average accuracy over the sampled responses.
The cited benchmark resources are https://huggingface.co/datasets/openlanguagedata/flores_plus and https://github.com/ByteDance-Seed/Seed-X-7B/tree/main/challenge_set.

4.2.1 DuPO Boosts LLM’s Performance on Various Tasks

In 4.2 Main Results, Table 1 shows Seed-X-7B-Instruct improving from 28.76 to 30.31 BLEU, 86.96 to 89.09 COMET, and 72.62 to 74.57 BLEURT. Figure 2 reports an average human score of 3.67, compared with 3.65 for GPT-4o, 3.64 for DeepSeek-R1-0120, and 3.44 for Google Translate; no uncertainty estimates accompany these point comparisons.

Table 2 reports Qwen3-4B improving from 77.2% to 83.6% average mathematics accuracy and OpenReasoning-Nemotron-7B from 83.9% to 90.3%, both gains of 6.4 percentage points.
DeepSeek-R1-Distill-Qwen-7B rises from 59.5% to 64.4%; the 1.5B model’s reported average rises from 35.8% to 39.7%.
The 1.5B DuPO row lists 72.5%, 30.0%, and 26.7%, whose arithmetic mean is approximately 43.1%, rather than the stated 39.7%. Its reported average gain therefore cannot be reconciled with that row.
OpenReasoning-Nemotron-7B with DuPO reaches 90.3%, below DeepSeek-R1-0528’s listed 92.8%; the table supports improvement over its own baseline.

Multilingual translation performance across 756 translation directions in 28 languages.

Human evaluation scores on Seed-X-Challenge for 14 language directions.

Mathematical reasoning performance on AMC23, AIME24, and AIME25.

4.2.2 DuPO Scales to Various Backbones Effectively

The architecture comparison evaluates LlaMA-3.1-8B and OctoThinker-8B-Hybrid-Base on AMC23 and MATH500, a mathematical problem-solving benchmark. The paper states that the models are fine-tuned with identical training data and settings.

LlaMA-3.1-8B improves from 8.1% to 32.1% average accuracy, compared with 19.0% for SimpleRL-Zoo, an answer-supervised reinforcement-learning baseline.
OctoThinker-8B-Hybrid-Base improves from 23.8% to 62.5%, with AMC23 rising from 5.0% to 55.0% and MATH500 from 42.6% to 70.0%.
The results extend the evidence to the LlaMA family; OctoThinker’s prior mathematics training is part of its starting condition.

DuPO performance on different backbone models.

4.2.3 DuPO Incentivizes Reasoning Capability on Base Model

DuPO is applied directly to Qwen3-4B-Base without a preliminary supervised fine-tuning (SFT) stage, training on example responses. Figure 3 tracks forward-task training accuracy and performance on unseen AMC23, AIME24, and AIME25 problems.

Forward accuracy increases from 15.2% to 56.5%, while AMC23 accuracy increases from 20% to 70%.
AIME24 and AIME25 also improve across the plotted checkpoints.
The figure reports one training trajectory without variability across repeated runs.

Training progress of DuPO on Qwen3-4B-Base.

4.2.4 DuPO Scales Reasoning during Inference without Training

At inference, DuPO generates candidate reasoning trajectories, uses each final answer to condition automatically constructed complementary questions, and selects the trajectory with the highest backward accuracy, the fraction of successful hidden-value reconstructions. Each dual question receives K = 8 sampling runs to estimate this score without additional parameter training.

Table 4 reports Qwen3-4B increasing from 68.4% to 77.7% average accuracy across AIME24 and AIME25, a 9.3 percentage-point gain.
DeepSeek-R1-Distill-Qwen-1.5B increases from 20.0% to 38.7%, a gain of 18.7 percentage points.
Qwen3-4B’s reranked average exceeds the listed DeepSeek-R1-0120 and Claude-Sonnet4-Thinking averages of 74.9% and 76.3%, respectively.
The gain requires additional candidate generation and reverse-task computation; Table 4 does not compare systems under a matched total inference budget.

Inference-time mathematical reasoning performance using DuPO backward accuracy for reranking.

4.3 Effects of Task Duality

The ablation removes unknown-component selection while keeping the stated experimental settings identical. Figure 4 displays filtered versus unfiltered average accuracies of 43.1% versus 39.5% for the 1.5B model and 84.7% versus 79.3% for Qwen3-4B.

The paper reports filtering benefits of 3.6 and 5.4 percentage points.
The filtered averages differ from Table 2’s 39.7% and 83.6%; Qwen3-4B’s AIME25 score is also 73.3% in Figure 4 versus 70.0% in Table 2.
The comparison supports a filtering benefit within this ablation, but the unexplained differences prevent combining its results with Table 2 as a single consistent measurement set.

Mathematical reasoning ablation of unknown-component selection for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B.

5 Conclusion

The conclusion identifies generalized duality as the main contribution: reconstructing selected input components supplies feedback for reinforcement-learning training and inference-time reranking. The experiments support performance improvements in translation and mathematics, but do not establish the broader claims of autonomous or cost-effective optimization.

6 Limitations

The authors identify computational overhead from selecting unknown mathematical components, limited evidence on substantially larger trained models, and untested application to open-ended or creative instruction-following tasks. These limitations bound the demonstrated applicability and scalability of DuPO.

Appendix

A Construction of Dual Question in Math Reasoning identifies numerical candidates while excluding subscripts, inequality constraints, common exponential bases, and function arguments. It replaces each eligible number with an identifier of the form Variable_{str}, where str is a randomly generated lowercase string, and constructs an inverse question conditioned on an answer. Filtering requires that exactly one answer in the sampled candidate pool enable successful reconstruction; this condition does not prove global mathematical uniqueness.
B Math RL Dataset Preparation starts with 1,815,942 public-dataset prompts, obtains 318,649 primal questions after deduplication, and generates 1,059,671 dual questions. Qwen2.5-7B-Instruct supplies 32 candidate answers per primal question for sampling and filtering, followed by another pass using Qwen3-4B with 8 candidates per question. The final retained training-set size is not specified.
C Experiment Details specifies a training batch size of 512, mini batch size of 32, sampling temperature of 1.0, 16 rollouts, sampled model responses, per prompt, learning rate of 1e-6, and gradient clipping of 1.0, a limit on gradient magnitude. Maximum input length is 2,048 tokens; maximum output length is 4,096 tokens for translation and 30,000 tokens for mathematics. The mathematics training output limit differs from the 32,000-token evaluation reasoning budget.
D Case Study illustrates reconstruction scoring with Table 5. For a geometry problem with circumradius 13, the radius of the circle through the triangle’s vertices, and inradius 6, the radius of its inscribed circle, candidate answer 468 receives 69.1% backward accuracy, while 108 and 312 receive 0%; the translation candidates have forward/backward BLEU scores of 45.85/82.07 and 28.65/16.11. These selected examples show agreement between reconstruction scores and candidate quality without measuring reliability across the full evaluation distribution.

Brief Thoughts

DuPO offers a concrete way to use a hidden input component as a checkable reward target, with filtering and architecture comparisons supporting its usefulness for learning. Practical assessment requires accounting for candidate generation, dual-question filtering, and reverse-task sampling costs, as well as reconciling the inconsistent mathematical results. The evidence supports reconstruction feedback in mathematics and translation; reliability across arbitrary tasks remains unestablished.

Case studies of DuPO on mathematical reasoning and machine translation.