Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions | Summary

Contents

Summary
1. Introduction
2. Marco Reasoning Datasets
3. Solution Space Expansion via MCTS
4. Reasoning Action Strategy
- 4.1. Action Selection
- 4.2. Reflection after Thinking
5. Experiments
6. Conclusions and Future Work
Appendix
Brief Thoughts

This article explains the key points of Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions.

2024-11-21 (arXiv)
Zhao, Yu, Yin, Huifeng, Zeng, Bo, Wang, Hao, Shi, Tianqi, Lyu, Chenyang, Wang, Longyue, Luo, Weihua, Zhang, Kaifu.
MarcoPolo Team, Alibaba International Digital Commerce
Paper
Github

Summary

Marco-o1 investigates how language models can solve problems with unclear answer standards, including colloquial translation. Compared with Qwen2-7B-Instruct under Chain-of-Thought (CoT) prompting, which requests intermediate reasoning steps, complete-step search increases English MGSM math-word-problem accuracy from 84.00% to 90.40%, while search using 32-token segments increases Chinese accuracy from 76.80% to 82.40%; tokens are the text units generated by the model.
The method combines full-parameter supervised fine-tuning, which updates all model weights using provided examples, with Monte Carlo Tree Search (MCTS), which explores alternative reasoning paths through a tree. Training uses 60,266 examples; during inference, or answer generation, search compares complete reasoning steps with smaller actions of 64 or 32 tokens and scores completed paths using confidence derived from token probabilities.
Search improves math accuracy, but the best action size differs across languages. Open-ended translation evidence consists of three illustrative cases, while search randomness, missing compute budgets, and limited reporting of reflection, a prompt to reconsider earlier reasoning, restrict broader conclusions.

1. Introduction

Inspired by OpenAI’s o1, the paper asks whether large reasoning models, language models designed for extended reasoning, can generalize beyond tasks with standard answers and readily quantified rewards. Its contributions combine CoT fine-tuning, confidence-guided MCTS, variable action granularity, prompted reflection, and translation case studies; the authors explicitly describe the model as work in progress with o1-like characteristics and performance below a fully realized o1 model.

The project homepage is https://github.com/AIDC-AI/Marco-o1.
Figure 1 illustrates a correct answer to the strawberry letter-counting question. The displayed reasoning omits the final “y”; the authors’ suggestion that this reflects confidence in skipping an obvious step remains a hypothesis.
The introduction reports improvements of +6.17% for English and +5.60% for Chinese. Table 2 instead implies absolute gains of 6.40 and 5.60 percentage points over Qwen2-7B-Instruct; the English discrepancy is unresolved.

A reasoning tree and model response for counting the letter “r” in “strawberry.”

2. Marco Reasoning Datasets

Marco-o1-CoT is trained on three datasets totaling 60,266 samples: 45,125 examples from the Open-O1 CoT Dataset (Filtered), 10,000 from the Marco-o1 CoT Dataset (Synthetic), and 5,141 from the Marco Instruction Dataset. The first supplies filtered reasoning patterns, the second supplies MCTS-generated reasoning paths, and the third is intended to preserve instruction-following ability.

Open-O1 filtering is described as heuristic and quality filtering, but the paper does not specify reproducible filtering rules.
Figure 2 separates supervised training from inference-time search and shows how action granularity, the size of each generated continuation, and confidence scoring enter the search procedure.

Sample counts for the three Marco Reasoning Datasets.

Marco-o1 training and inference pipeline with action strategies and confidence scoring.

3. Solution Space Expansion via MCTS

Each MCTS node represents a reasoning state, and each action is a model-generated continuation. A rollout, a continuation to a terminal answer, receives a confidence-based value that guides path selection and backup, the propagation of values through the tree.

The printed token-confidence equation is $c_i = \frac{\exp(p(t_i))}{\sum_{k=1}^{5}\exp(p(t_k))}$, where p(t_i) is the logarithm of the generated token’s probability and the denominator uses the top 5 predicted tokens at that position.
The rollout value is $v = \frac{1}{n}\sum_{i=1}^{n}c_i$, where n is the rollout’s token count. This is a proxy based on model confidence; the paper does not establish that it reliably measures answer correctness.
The prose refers to the generated token and “top 5 alternative tokens,” whereas the printed denominator contains five terms. This wording leaves the precise candidate-set construction unclear.

4. Reasoning Action Strategy

The reasoning action strategy changes where search can branch and adds a prompt for reconsidering a completed thought process. Action selection controls search resolution, while reflection asks the model to reevaluate its reasoning.

4.1. Action Selection

“Step as Action” generates a complete reasoning step at each branch, while “Mini-step as Action” uses continuations of 32 or 64 tokens. Smaller actions permit more detailed branching, but the paper identifies computational cost and reward-design difficulty as obstacles to token-level search.

The experiments do not establish a universally superior granularity: complete steps perform best on English, while 32-token mini-steps perform best on Chinese.
Figures 5 and 6 show opposite outcomes for finer search. These examples demonstrate that smaller actions can help or fail on individual problems, without establishing what determines the better action size.

4.2. Reflection after Thinking

Reflection appends the exact phrase “Wait! Maybe I made some mistakes! I need to rethink from scratch.” at the end of each thought process. The authors report that approximately half of the difficult problems initially answered incorrectly become correct after reflection.

The paper does not provide the size or selection criteria of this problem set, a controlled results table, or a separate ablation, a comparison that isolates reflection’s contribution. The reported recovery rate therefore has limited experimental context.

5. Experiments

The experiments evaluate math reasoning quantitatively and colloquial translation qualitatively. The quantitative comparisons examine the base model, the fine-tuned model, and three search configurations; the translation section presents selected outputs alongside Google Translate.

5.1. Setup

Full-parameter fine-tuning of Qwen2-7B-Instruct produces Marco-o1-CoT, which then supplies the model for all three MCTS configurations: complete steps, 64-token mini-steps, and 32-token mini-steps. Every model receives a CoT prompt during testing on the English (En) and Chinese (Zh) MGSM subsets.

The base-model URL is https://huggingface.co/Qwen/Qwen2-7B-Instruct.
The paper does not report training hyperparameters, search rollout budgets, detailed sampling settings, inference latency, or uncertainty estimates. These omissions limit reproduction and comparisons at matched computational cost.

5.2. Main Results

Table 2 reports English/Chinese accuracy of 84.00%/76.80% for Qwen2-7B-Instruct, 85.60%/71.20% for Marco-o1-CoT, 90.40%/80.00% for complete-step MCTS, 88.40%/80.40% for 64-token mini-step MCTS, and 87.60%/82.40% for 32-token mini-step MCTS. Every search configuration improves on Marco-o1-CoT in both languages, while fine-tuning alone improves English and reduces Chinese accuracy.

The authors attribute the Chinese decline after fine-tuning to English CoT training data, but they do not provide a controlled multilingual-data ablation.
Figure 3 visualizes the language-dependent ranking of search configurations. The authors acknowledge substantial randomness from the confidence-based reward and do not identify one action strategy as generally superior.

English and Chinese MGSM accuracy for the base model, Marco-o1-CoT, and three MCTS configurations.

Bar-chart comparison of the main English and Chinese MGSM results.

Test@N is the percentage of problems answered correctly at least once across N separate guesses, evaluated at Test@1, Test@8, and Test@32. Complete-step MCTS reaches 90.40%/99.20%/99.20% on English, compared with 85.60%/97.60%/99.20% for Marco-o1-CoT; on Chinese, 32-token mini-step MCTS reaches 82.40%/93.20%/96.80%, compared with 71.20%/93.60%/96.40%.

The best English Test@32 result is 99.60% with 64-token mini-steps, versus 96.00% for the base model. The best Chinese Test@32 result is 96.80% with 32-token mini-steps, versus 92.40% for the base model.
Search has a clearer advantage at Test@1 than at larger guess counts. Because each search-based guess can itself involve multiple continuations, these results do not establish an advantage at equal inference compute.
Test@N measures whether a correct answer appears among the guesses; it does not demonstrate that the system can select that answer without access to correctness information.

MGSM Test@1, Test@8, and Test@32 solve rates, counting a problem as solved when at least one guess is correct.

The math case studies illustrate how alternative paths can correct a specific error, while also showing that smaller actions can fail. Figure 4 contrasts an incorrect 120-minute download calculation with the correct 160 minutes; Figure 5 contrasts an incorrect $195,000 renovation profit with the correct $70,000; Figure 6 contrasts an incorrect 31-kitten answer with the correct 40.

In Figure 4, complete-step MCTS accounts for restarting the entire 200 GB download after a 20-minute system restart, rather than downloading only the remaining 120 GB.
In Figure 5, 32-token mini-steps apply the 150% increase to the $80,000 purchase price, then subtract the $130,000 total investment.
In Figure 6, complete-step MCTS correctly adds 7 adopted kittens, 21 kittens from Patchy, and 12 from Trixie, whereas the 64-token mini-step output contains inconsistent counting.

Marco-o1-CoT and complete-step MCTS responses to an interrupted-download problem.

Complete-step and 32-token mini-step MCTS responses to a house-renovation profit problem.

64-token mini-step and complete-step MCTS responses to a kitten-counting problem.

5.3. Case Study on Translation Tasks

Three Chinese-to-English cases illustrate reasoning about colloquial meaning. Figure 7 interprets a literal “stepping on feces” shoe description as a comfortable sole; Figure 8 interprets a clothing description as Korean style; Figure 9 recognizes repeated “太” as emphasis rather than the word “madam.”

Figures 8 and 9 explicitly label the model output as Marco-o1-CoT and compare it with Google Translate. They therefore do not isolate the contribution of MCTS to translation quality.
The examples support specific improvements in contextual interpretation. No systematic translation benchmark, aggregate metric, blinded human evaluation, or compute-scaling experiment is reported, so broad superiority in open-ended translation remains unestablished.

A Marco-o1 reasoning path for translating a colloquial shoe-comfort description.

Google Translate and Marco-o1-CoT outputs for a colloquial clothing description.

Google Translate and Marco-o1-CoT outputs for an emphatic colloquial product recommendation.

6. Conclusions and Future Work

The authors conclude that CoT fine-tuning and MCTS can improve reasoning, with action granularity influencing the paths explored. They propose Outcome Reward Modeling (ORM), which scores final outcomes, and Process Reward Modeling (PRM), which scores intermediate reasoning, to improve the search reward; reinforcement learning, training guided by rewards, is also identified as future work.

These reward-model and reinforcement-learning proposals are not evaluated components of the reported system. Their expected benefits remain future-work claims.

Appendix

The PDF contains no explicitly titled appendix. Figures 4–9 appear on later pages, including pages after the references, but are discussed as evidence for the main-body experiments.

Brief Thoughts

The strongest result is the consistent improvement from search over Marco-o1-CoT across both tested languages, with different action granularities producing different accuracy rankings. Evidence for open-ended reasoning is limited to selected translation cases; missing compute budgets, uncertainty estimates, and controlled reflection results leave efficiency, stability, and component contributions unresolved.