Phi-4 Technical Report | Summary

This article explains the key points of Phi-4 Technical Report.

• 2024-12-12 (arXiv)
• Abdin, Marah, Aneja, Jyoti, Behl, Harkirat, Bubeck, Sébastien, Eldan, Ronen, Gunasekar, Suriya, Harrison, Michael, Hewett, Russell J., Javaheripi, Mojan, Kauffmann, Piero, et al.
• Microsoft Research
• Paper

Read this article in Korean

Summary

• Improving a compact language model’s reasoning requires training data that teaches useful solution steps while retaining factual knowledge. Under simple-evals, an evaluation framework fixing prompts, answer extraction, and sampling settings, at temperature 0.5, the 14-billion-parameter phi-4 surpasses its teacher GPT-4o on GPQA, a graduate-level science question-answering benchmark, by 56.1 to 50.6 and on MATH, a competition mathematics benchmark, by 80.4 to 74.6.
• The recipe combines synthetic data, model-generated training material, with filtered human-generated or otherwise non-synthetic sources. Approximately 10T tokens, the text units processed during training, are used for pretraining, the initial broad training stage, followed by midtraining to extend the input context, Supervised Fine-Tuning (SFT) on selected example responses, and Direct Preference Optimization (DPO) on preferred and rejected outputs.
• Pivotal Token Search (PTS), a method that identifies individual tokens whose selection sharply changes eventual answer correctness, supplies targeted preference pairs for the first DPO stage. Fresh November 2024 American Mathematics Competitions (AMC) tests support the mathematics results, but factual knowledge, strict instruction following, and some long-context tasks remain weaknesses; the report also documents evaluation ambiguities and incomplete protection against benchmark contamination, the presence of evaluation material in training data.

1 Introduction

The report attributes phi-4’s improvements primarily to data generation, data mixture design, and post-training, with relatively small architectural changes from phi-3-medium. Its three development pillars are synthetic data for pretraining and midtraining, curation of high-quality non-synthetic data, and improved post-training datasets and preference pairs.

• Synthetic generation uses multiple model roles, iterative critique and revision, and instruction reversal, which creates instructions from existing outputs such as code.
• Table 1 shows strong mathematics and science results alongside weaker factual question answering and strict instruction following.

Phi-4 leads the listed models on GPQA and MATH while trailing substantially on factual answering and strict instruction following. The comparison establishes task-specific strengths under the reported evaluation procedures; the first benchmark block uses simple-evals at temperature 0.5.

1.1 Addressing Overfitting and Data Contamination

The authors address contamination through stronger filtering, original internal questions, and evaluation on November 2024 AMC-10 and AMC-12 problems released after training data collection. Figure 1 reports phi-4 at 91.8 out of 150, compared with GPT-4o at 77.9 and Gemini Pro 1.5 at 89.8, at temperature 0.5.

• The figure caption reports averages over four tests on 100 runs with error bars of 2σ, twice the estimated standard deviation; Appendix C specifies 10 independent generations per question, without explaining how these sampling descriptions relate.
• QwQ-32B-Preview, which generates long step-by-step reasoning during inference, scores 124.5 in this setting but uses 4X more tokens and more than twice the parameters. The report therefore places it in a different cost and latency class.
• The test dates provide evidence against direct training overlap, although Appendix C discloses that two other final candidates were evaluated before the final model’s score was measured.

Phi-4 scores 91.8 out of 150, ahead of Gemini Pro 1.5 at 89.8 and GPT-4o at 77.9 on newly released competition tests. The caption specifies 100 runs and 2σ error bars, while Appendix C specifies 10 independent generations per question without explaining the relationship.

2 Approach to Data

Pretraining relies heavily on synthetic datasets generated through several complementary techniques. Filtered non-synthetic sources serve both as direct training material and as seeds, source passages or questions used to initiate synthetic generation.

2.1 Purpose of Synthetic Data

The authors argue that synthetic text can make reasoning easier to learn through next-token prediction, predicting each text unit from its predecessors, by presenting intermediate steps in generation order. They also argue that rewriting source knowledge into assistant-style text brings pretraining examples closer to the contexts encountered during inference.

• The stated generation principles are diversity, nuance and complexity, accuracy, and Chain-of-Thought, explicit step-by-step reasoning.
• This section proposes explanations for the data recipe without independently quantifying each mechanism.

2.2 Synthetic Data for Pretraining and Midtraining

The team created 50 broad synthetic dataset types totaling about 400B unweighted tokens, using different seeds and multistage prompting procedures. Generation combines source selection, rewriting into exercises or discussions, self-revision, instruction reversal, and validation.

• Source passages are filtered first for educational value and then for factual and reasoning content. Question seeds are screened through multiple independent answers, excluding unanimous agreement and entirely inconsistent answers; another pipeline converts deduction chains in source material into question-answer pairs.
• Plurality answers, the most frequently generated answers, replace unavailable reference answers during rejection sampling, which retains generated responses meeting the selected answer criterion.
• Instruction reversal derives task prompts from existing code and retains pairs with high fidelity between original and regenerated code; execution and tests validate synthetic code where appropriate.

2.3 Curation and Filtering of Web and Q&A Data

The authors collected tens of millions of non-synthetic problems and solutions and report that original questions were substantially more effective than synthetic questions in their ablations. They replaced unreliable source answers with generated answers supported by majority voting, while filtering questions and solutions against evaluation sets.

• Small classifiers trained on approximately 10^6 model-generated annotations select high-quality web documents. A separate pipeline increases coverage of arts, history, travel, culture, and entertainment, which the initial filtering underrepresented.
• Multilingual processing uses a language identification model based on fastText, a text classification tool, to categorize documents into 176 languages before quality filtering.
• Custom extraction preserves equations, code, tables, and forum structure across web pages and document formats. The report stresses that errors in seed material can propagate into synthetic documents.

2.4 Post-Training datasets

Post-training datasets combine public and synthetic prompts with multiple generated responses. Model-based evaluation selects responses for SFT, while rejection sampling and model evaluation create preferred and rejected pairs for DPO, including pairs that isolate pivotal tokens.

3 Pretraining details

Phi-4 uses a decoder-only transformer, an architecture that generates text from preceding tokens, with 14B parameters and an initial context length of 4096. Relative to phi-3-medium, it uses the tiktoken tokenizer, which converts text into model tokens, with a padded vocabulary of 100,352 including unused tokens, and full attention over 4K tokens instead of a 2K sliding window, allowing each token to use the full preceding context rather than a restricted recent portion.

• Pretraining uses approximately 10T tokens, linear learning-rate warm-up and decay, peak learning rate 0.0003, weight decay 0.1, and global batch size 5760.
• Base-model evaluation uses internal log-likelihood scoring of candidate answers and few-shot prompts containing task examples, because strict answer formatting without examples can obscure pretrained capabilities.
• Table 2 reports gains on most evaluated tasks, including MMLU, a broad multitask language-understanding benchmark, its more challenging MMLU pro variant, and code generation. TriviaQA, a factual question-answering benchmark abbreviated TQA in the tables, declines.

Both the 4K and 16K checkpoints improve most reported base-model evaluations relative to phi-3-medium, while TriviaQA declines by 0.7 and 1.5 respectively. The 16K checkpoint retains gains over the predecessor but does not consistently exceed the 4K checkpoint.

3.1 Data Composition in Pretraining

At a fixed training-token budget, phase 2 experiments with 7B and 14B models favor 12 repetitions of a fixed synthetic dataset over four repetitions supplemented by more unique web tokens. However, 13B ablation models, used to compare data choices, trained without raw web data improve several reasoning and coding scores while losing substantial factual question-answering performance.

• Figure 2 shows the synthetic repetition advantage on 5-shot MMLU, with five examples supplied in the prompt, without an observed deterioration over the plotted training interval.
• Relative to phi-3-medium, the synthetic-only model loses 14.8 on TriviaQA; mixing synthetic data equally with web rewrites, generated reformulations of web content, reduces the deficit to 7.7.
• These experiments support repeated curated synthetic training within the tested settings, while the knowledge deficits motivate retaining non-synthetic sources.

For both 7B and 14B models, the 12-epoch synthetic-data runs achieve higher MMLU scores than the four-epoch alternatives under the same training-token horizon. Within these experiments, replacing additional unique web tokens with more repetitions of the fixed synthetic dataset improves the measured outcome.

The synthetic-only 13B model gains 12.1 on Human-Eval but loses 14.8 on TriviaQA relative to phi-3-medium. Using equal shares of synthetic data and web rewrites reduces the factual deficit to 7.7 while retaining reasoning and coding gains; neither variant includes raw web data.

3.2 Data Mixture

Data mixture selection uses shorter 1T-token experiments, largely at 7B scale, based on the authors’ observed ranking consistency across training horizons and model sizes when mixtures differ sufficiently and data sources have not saturated. The final pretraining allocation is 15% web, 15% web rewrites, 40% synthetic, 20% code data, and 10% acquired sources.

• Table 4 varies synthetic, filtered web, and web rewrite allocations within 75% of the ablation token budget while holding the remaining 25% constant. A uniform allocation scores 2.2 lower on the reported average than the final mixture.
• Synthetic-heavy alternatives improve the average by 0.8 or 0.4, but the selected mixture retains knowledge-heavy material to balance capabilities. The authors report that much of this gap closes after post-training.
• Table 5 lists approximately 290B unique synthetic tokens repeated for 13.8 epochs, complete passes over that source, compared with 1.3T unique web tokens repeated for 1.2 epochs. Code data includes both synthetic and raw code.

A uniform allocation across the three varied categories loses 2.2 on the reported average, while synthetic-heavy allocations gain 0.8 or 0.4. Greater web allocation benefits TriviaQA, illustrating the competing reasoning and knowledge results considered when selecting the final mixture.

Synthetic data receives 40% of training despite containing only 290B unique tokens, producing 13.8 epochs of exposure. Web data contains 1.3T unique tokens but receives 15% of training and 1.2 epochs, showing how training allocation differs from each source’s unique corpus size.

3.3 Midtraining Details

Midtraining increases the supported context from 4K to 16K using 250B tokens. The mixture contains 30% newly curated longer-context material and 70% replayed pretraining material, with a tenfold reduction in maximum learning rate and the base frequency of rope position encoding, the model’s positional representation, increased to 250K.

• Naturally long documents outperform artificially packed shorter samples in the reported ablations. The pipeline selects non-synthetic samples above 8K, upweights those at least 16K long, and creates synthetic sequences longer than 4K.
• HELMET, an evaluation suite covering multiple long-context tasks, is evaluated at 8K and 16K with results averaged across five runs per category.
• At 16K, phi-4 scores 99.0 on key-value recall, retrieving a value given its key, but 57.1 on retrieval-augmented question answering from retrieved documents, versus GPT-4o’s 66.7. Recovering an isolated value and answering from documents therefore show different performance.

Near-perfect key-value recall coexists with weaker document-based answering and summarization. At 16K, phi-4 scores 77.0 on in-context learning, learning a task from examples in the prompt, but trails GPT-4o on every listed category.

4 Post-Training

Post-training converts the pretrained model into an assistant through one SFT round, one DPO round using pivotal-token pairs, and one DPO round using full-length preference pairs. Conversations use chatml, the role-delimited conversation format shown in the report.

4.1 Supervised Fine-Tuning

SFT uses around 8B tokens at learning rate 10^−6. Training covers mathematics, coding, reasoning, conversation, model identity, safety, and multilingual data for 40 languages, all in chatml format.

4.2 Direct Preference Optimization

The first DPO round uses PTS-generated preference pairs, while the second uses approximately 850k full-response pairs labeled by GPT-4o. For the second round, responses from GPT-4o, GPT-4t, and phi-4 are compared by accuracy or by the average of accuracy, style, and detail.

• Table 7 lists 132,859 generic multiple-choice samples, 76,552 mathematics samples, 16,080 Python samples, 21,806 samples in other programming languages, and 3,000 unknown-answer and safety samples for pivotal-token DPO.
• Table 8 lists 266,000 overall-preference pairs, 532,000 accuracy-preference pairs, and 43,842 unknown-answer and safety pairs for judge-guided DPO.
• Both stages include data intended to reduce unsafe behavior and hallucinations, unsupported or fabricated answers.

Generic multiple-choice questions form the largest pivotal-token training category at 132,859 samples, followed by mathematics at 76,552. Together with the programming categories, these data match the method’s emphasis on tasks with accessible correctness checks.

Accuracy-based pairs number 532,000, twice the 266,000 overall-preference pairs, alongside 43,842 unknown-answer and safety pairs. The second stage combines correctness-oriented supervision with broader response-quality judgments.

4.3 Pivotal Token Search

PTS estimates the probability of eventual correctness after a response prefix by sampling continuations and checking them with an oracle, a correctness check such as known mathematical answers or code tests. It recursively subdivides a completion to locate tokens associated with a sharp change in success probability, then constructs accepted and rejected single-token continuations sharing the same prefix.

• Questions are restricted to 0.2 ≤ p(success) ≤ 0.8, where p(success) denotes the probability of a correct answer, to focus search on tasks of intermediate difficulty.
• Figure 3 distinguishes pivotal tokens from merely unlikely tokens: selecting “negative” changes estimated success from 0.42 to 0.93 in the illustrated mathematics solution.
• Figure 4’s search can miss pivotal tokens when opposing changes cancel within a segment; the authors state that it finds all pivotal tokens when success probabilities are near-monotone, changing mostly in one direction. Figure 5 shows that a rejected continuation can be mathematically valid but less reliable for subsequent generation.
• The method requires repeated sampled continuations and reliable correctness checks. The report relates it to automated process supervision, training signals for intermediate solution decisions, but does not provide a comprehensive generation-cost comparison.

Continuation sampling shows that “negative” raises estimated success from 0.42 to 0.93, whereas a later token reduces it from 0.95 to 0.71. The example motivates supervising consequential choices individually because an ultimately correct response can contain choices that reduce the reliability of subsequent generation.

The recursive procedure compares success probabilities at segment boundaries and subdivides segments until it isolates a token or the change falls below the threshold. Each probability estimate requires sampled continuations and correctness checking; the pseudocode recommends caching estimates to avoid repeating that work.

The mathematics, physics, and Python examples share a response prefix and contrast continuations whose underlined tokens supply the preference pair. The mathematics example shows that preferences can distinguish downstream reliability between valid solution approaches.

4.4 Hallucination mitigation

Dedicated SFT examples and DPO pairs teach the model to decline questions it is unlikely to answer correctly. In Figure 6, incorrect SimpleQA responses, measured on a benchmark of short factual questions, fall from 90.0% for the base model to 15.8% for the final model, while unattempted questions rise from 3.2% to 81.1%.

• The correct-answer share also falls from 6.8% to 3.0%.
• The result demonstrates greater abstention on this benchmark, with fewer incorrect answers and fewer correct answers; it does not demonstrate an increase in factual knowledge.

Incorrect SimpleQA answers decrease from 90.0% to 15.8%, chiefly as unattempted questions increase from 3.2% to 81.1%. Correct answers also decrease from 6.8% to 3.0%, so the reduction in errors accompanies lower answer coverage.

4.5 Post-Training Ablation

Table 9 compares SFT, pivotal-token DPO, judge-guided DPO alone, and both DPO stages. Pivotal-token DPO raises GPQA from 47.3 to 53.6 and MATH from 77.1 to 80.5; the final model reaches 56.1 and 80.4, respectively.

• ArenaHard, a response-generation benchmark evaluated by a model judge, rises from 56.7 after SFT to 66.5 after pivotal-token DPO, 69.8 with judge-guided DPO alone, and 75.4 with both stages.
• The stages are complementary on several measures, but gains are uneven: DROP, a reading-comprehension benchmark, falls from 86.1 after stage 1 to 75.5 finally, and IFEval, a strict instruction-following evaluation, falls from 66.2 after SFT to 63.0.

The full sequence produces the highest GPQA and ArenaHard results among the compared variants, supporting complementary effects from the two DPO stages. DROP declines sharply after the second stage, and IFEval remains below its SFT value, revealing task-specific regressions.

5 Benchmarking Considerations

The report identifies four evaluation concerns: residual contamination despite filtering, narrow skill coverage, stylistic bias in model-judged generation, and multiple-choice success through guessing or pattern matching. PhiBench, an internal benchmark of team-written original prompts, broadens coverage to debugging, code explanation, proof-error identification, and related tasks.

• Detailed judge notes emphasize accuracy, logical structure, and task requirements. The authors report improved scoring consistency without giving a numerical consistency analysis here.
• PhiBench guides data mixture and post-training hyperparameter choices, so it serves as a development instrument as well as a test of performance on original questions.
• The authors acknowledge that matching contiguous word sequences does not reliably remove all paraphrased benchmark material.

6 Performance on Key Benchmarks

The authors report that phi-4 exceeds Qwen-2.5-14B-Instruct on 9 of 12 benchmarks in their comparison, with strong science, mathematics, and code-generation results. Table 1 includes MATH at 80.4 versus 75.6 and HumanEval, a code-generation evaluation, at 82.6 versus 72.1.

• The first benchmark block uses simple-evals with its prompts, extraction rules, and temperature 0.5. Several additional benchmarks use internal prompting and extraction.
• Weak results include SimpleQA at 3.0, DROP at 75.5, and IFEval at 63.0, compared with Qwen-2.5-14B-Instruct at 5.4, 85.5, and 78.7.
• The paper attributes some SimpleQA score loss to deliberate abstention and accepts strict instruction following as a substantive weakness. It also suggests that strict output formatting explains some lower Llama scores compared with the model developer’s reported results.

7 Safety

Safety development combines post-training alignment, training intended to encourage desired behavior, adversarial testing, and automated evaluations across dozens of Responsible AI (RAI) harm categories. Training draws on helpfulness and harmlessness preference datasets together with modified and internally generated safety data.

7.1 RAI Benchmarks

The internal RAI evaluation uses GPT-4o to simulate and assess multi-turn conversations in five categories. Grounding, whether an answer is supported by its prompt, is scored from 0 to 5; other categories use harm severity from 0 to 7 and defect rates, the share of samples meeting a specified severity threshold.

• Table 10 reports phi-4 at 0.121 for third-party content harms and 0.073 for jailbreaks, attempts to bypass safety restrictions, lower than all listed comparison models on these measures.
• Results are mixed across categories: harmful-content continuation is 0.036, higher and therefore worse than every listed comparator, while grounding is 4.619 versus phi-3-medium’s 4.787.
• These results support category-specific safety comparisons under the internal evaluation; they do not show improvement across every measure.

Phi-4 has the lowest reported third-party content-harm and jailbreak defect rates among the listed models, but the highest harmful-content continuation rate at 0.036. Boldface identifies phi-4 values rather than category winners.

7.2 Red Teaming

The Microsoft AI Red Team conducted a two-week red-teaming exercise, adversarial testing intended to expose safety and security failures, covering ordinary and adversarial users in single-turn and multi-turn settings. Identified risky behaviors prompted additional safety post-training; tested attacks included jailbreaks, encoded prompts, and attacks distributed across conversation turns.

• Adversarial suffixes, text appended to prompts to induce unwanted behavior, generated against phi-3-medium did not transfer to phi-4 in the reported exercise.
• The authors call for further testing across additional scenarios and harm categories. This section provides qualitative findings without numerical attack-success rates.

8 Weaknesses

Remaining weaknesses include fabricated factual biographies, failure to follow detailed formatting instructions, elementary reasoning errors, and unnecessarily elaborate responses. The report notes that phi-4 can incorrectly judge 9.9 smaller than 9.11 and that fine-tuning emphasized single-turn queries.

• The authors link formatting weaknesses partly to training that prioritized question answering and reasoning over strict instruction following.
• Bias, inappropriate generation, and safety failures remain despite mitigation.

Appendix

• A Post-Training Dataset Details describes SFT that selects answers or refusals according to estimated model answerability, and DPO that prefers correct answers over refusals and refusals over incorrect answers when those outcomes occur. It also supplies synthetic unanswerable questions, refusal and answer-checking prompts, and the judge-guided DPO rubric.
• B Data Processing describes n-gram decontamination using direct matches and overlap ratios, with common-phrase exceptions to avoid irrelevant removals. The algorithm leaves overlap thresholds unspecified, and matching word sequences cannot reliably remove all rephrased benchmark material.
• C AMC Evaluation Details describes inference and grading on competition questions released after training data collection. The final model was chosen on other factors after the other candidates’ performance had been seen, but before its own score was measured. The relationship between the figure’s and appendix’s sampling descriptions remains unexplained.
• D Synthetic generation examples covers metadata-guided selection, self-revision, conversation generation, fill-in-the-middle tasks and rewriting AgentKit trajectories into self-contained reasoning. The report claims internal improvements from AgentKit data without a quantitative ablation in this appendix.

Brief Thoughts

The clearest evidence for the training recipe comes from fixed-budget experiments favoring repeated curated synthetic data and post-training comparisons showing gains from pivotal-token preferences. The synthetic-only factual-knowledge deficit also explains why the final recipe retains non-synthetic sources. PTS offers a concrete way to turn correctness-checked continuation outcomes into individual-token preferences. Its practical cost remains difficult to assess without a fuller accounting of continuation sampling. Fresh AMC questions strengthen the mathematics evidence, while the unexplained run-count descriptions and disclosed evaluation of other final candidates qualify how independently the test was used. The reported results favor separate assessment of reasoning, factual answer coverage, formatting compliance, and multi-turn behavior because improvements in one measure can accompany regressions in another.