SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers | Summary

This article explains the key points of SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers.

• 2025-07-28 (arXiv), 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: MATH-AI
• Manem, Chaitanya, Brahma, Pratik Prabhanjan, Mishra, Prakamya, Liu, Zicheng, Barsoum, Emad.
• Advanced Micro Devices, Inc. (AMD)
• Paper
• Github

Read this article in Korean

Summary

• Training large language models (LLMs) to solve difficult mathematics problems is constrained by the shortage of challenging, reliable training examples. Adding 500 SAND-Math examples to LIMO, an 817-example curated training corpus, raises Qwen2.5-32B-Instruct accuracy on AIME25, the 2025 American Invitational Mathematics Examination benchmark, from 44.50% to 48.89%, exceeding augmentation with the synthetic mathematics dataset MetamathQA by 17.85 percentage points under the same training setup.
• SAND-Math generates problems without a seed dataset, meaning an existing collection of questions to modify, and retains 8,842 problems after filtering for answer agreement, duplication, benchmark overlap, solver difficulty, and web similarity. Its Difficulty Hiking step rewrites problems using an additional theorem and supporting concept; one iteration increases mean model-assigned difficulty from 5.02 to 5.98.
• With 1,500 examples added to LIMO, Difficulty Hiking improves average benchmark accuracy from 72.94% to 74.39% and AIME25 accuracy from 46.38% to 49.23% relative to non-hiked data. The experiments cover one fine-tuned model, small training subsets, and one hiking iteration; conflicting filtering statistics and model-based correctness checks leave reproducibility and data-quality questions unresolved.

1 Introduction

The paper motivates automated generation of difficult mathematics data by contrasting limited manual competition-problem curation with synthetic methods that mainly modify existing benchmark questions. It hypothesizes that strong models have metacognitive abilities, meaning an implicit understanding of what makes problems difficult, that can support generation from minimal prompts. The experiments test the usefulness of the resulting data without directly establishing this proposed mechanism.

2 SAND-Math Pipeline

SAND-Math, expanded as Synthetic Augmented Novel and Difficult Mathematics problems and solutions, combines generation, successive filtering, and problem rewriting. Its stages screen for solvability, agreement between sampled answers, redundancy, evaluation-set overlap, difficulty for a target solver, and similarity to publicly searchable content.

The diagram separates teacher generation, answer-agreement checking, redundancy removal, solver-based selection, web novelty checking, and Difficulty Hiking. Its feedback path allows repeated hiking, whereas the reported experiments evaluate one hiking iteration.

2.1 Question and Solution Generation

A teacher model, which generates the training questions and solutions, first produces a problem with a provisional solution to encourage solvability, then separately samples \(k\) solutions for that problem. Each solution contains a reasoning trace, the step-by-step derivation, and a final answer; the initial collection contains 23,437 question-and-solution sets.

2.2 Solution Correctness Filtering

Self-consistency filtering retains a problem only when all independently sampled final answers agree, reducing the collection to 17,578 examples, reported as approximately 74% of the original yield. One agreeing reasoning-and-answer pair is sampled for training. Agreement provides a screening signal but does not independently establish that the answer or derivation is correct.

2.3 De-duplication and Decontamination

Semantic de-duplication, which removes questions with similar meaning, uses the semhash filtering framework with minisilab/potion-base-8M at a 0.99 similarity threshold and removes 1,293 examples, reported as 7.3%. Decontamination, the removal of overlap with evaluation problems, retrieves the top-5 benchmark candidates per question for a model judge to assess and removes another 4 questions, leaving 16,281 examples.

2.4 Difficulty Filtering and Rating

Difficulty filtering retains only questions that the solver model Qwen2.5-32B-Instruct answers incorrectly, producing 9,211 examples at a reported 56.6% retention rate. Separately, Llama-3.3-70B-Instruct rates problems on a 1–10 scale using a rubric anchored to Art of Problem Solving (AoPS) competition references. These model-assigned scores support distribution analysis and Difficulty Hiking.

2.5 Novelty Filtering

Each retained question becomes a web-search query, and a sentence embedding model, which maps text into vectors for semantic comparison, measures its maximum similarity to retrieved snippets. Section 2.5 specifies the top-10 results and discards questions above \(\tau = 0.85\), removing 4% and leaving 8,842 problems; Appendix I instead specifies \(K = 20\) search results.

2.6 Difficulty Hiking: Systematically Increasing Problem Complexity

Difficulty Hiking asks the teacher to rewrite an existing question using its difficulty rating, a theorem from the same mathematical branch, and a concept from another domain. One iteration shifts mean rated difficulty from 5.02 to 5.98. The model-rated distribution places SAND-Math above the evaluated earlier synthetic datasets, with overlap with curated difficult corpora; Appendix K notes that rewrites often incorporate only one requested transformation.

The violin plots show distributions of model-assigned difficulty ratings. The 500-example SAND-Math subset concentrates at higher ratings than MetaMathQA and OpenMathInstruct-2, with overlap with LIMO and Open-R1; the plot does not independently measure human problem-solving difficulty.

3 Experiments and Results

Implementation Details: DeepSeek-R1 generates questions at temperature 0.8, produces \(k = 2\) solutions at temperature 0.6, and performs Difficulty Hiking; temperature controls randomness during sampling. Llama-3.3-70B-Instruct judges consistency, contamination, and difficulty, with difficulty scores averaged over 3 runs. Web novelty checking uses Alibaba-NLP/gte-Qwen2-7B-instruct embeddings and a self-hosted SearXNG search service accessed through LangChain.

Experimental Design: All comparisons fine-tune Qwen2.5-32B-Instruct with full-parameter supervised fine-tuning (SFT), training all model parameters on example solutions, for 10 epochs at a learning rate of 5e-6. Training uses the LLaMA-Factory framework and DeepSpeed ZeRO-3, a memory-saving distributed training strategy, on one node with 8× AMD Instinct MI300X graphics processing units (GPUs); generation uses the same node configuration.

• Standalone comparisons use 500 examples, while augmentation comparisons add 500 examples to the 817-example LIMO corpus; the Difficulty Hiking ablation adds 1,500 examples.
• Evaluation reports pass@1, the accuracy of a single sampled answer, estimated using 16 samples per problem at temperature 0.7 for AIME 2024/2025 and AMC, a mathematics competition benchmark. MATH500, a 500-problem mathematics benchmark, uses greedy decoding, which selects the highest-probability next token, at temperature 0.

Results: Standalone SAND-Math training achieves a 69.10% average, below the 70.27% obtained with openr1_math, the OpenR1 mathematics comparison subset. Adding 500 SAND-Math examples to LIMO reaches 73.32%, compared with 71.50% for LIMO alone, 72.51% for LIMO + openr1_math, 45.23% for LIMO + MetamathQA, and 48.50% for LIMO + OpenmathInstruct, another synthetic mathematics dataset.

• The 17.85-point AIME25 advantage compares SAND-Math augmentation at 48.89% with MetamathQA augmentation at 31.04%; the gain over LIMO alone is 4.39 points.
• These results support SAND-Math augmentation under the reported sampling and training conditions. The large declines after adding some comparison datasets do not establish their usefulness under other training settings.

Equal-size 500-example comparisons show that SAND-Math scores below openr1_math when used alone but gives the highest reported average when added to LIMO. The AIME25 comparison of 48.89% against 31.04% for MetamathQA accounts for the 17.85-point headline; comparison with LIMO alone gives a 4.39-point gain.

The Difficulty Hiking ablation holds the added sample count at 1,500: Base, the non-hiked data, achieves 72.94% average accuracy; DH, the Difficulty Hiked version, achieves 74.39%; and DH_w_LF, the hiked version with a 32k length filter, achieves 74.08%. DH improves AIME25 from 46.38% to 49.23%, AIME24 from 59.09% to 60.55%, AMC24 from 92.71% to 93.17%, and MATH500 from 93.6% to 94.6%.

• The length-filtered variant matches DH on AIME25 and slightly improves AIME24 and AMC24, but lowers MATH500 to 93%. Its lower average does not establish that longer solutions help every task.
• The appendices provide model-judged solution-quality and data-size analyses. The main training comparisons report no uncertainty estimates.

With 817 LIMO examples and 1,500 added examples in each augmentation condition, Difficulty Hiking raises the average from 72.94% to 74.39%. The length-filtered variant reaches 74.08%: it slightly improves AIME24 and AMC24 over standard DH but lowers MATH500, so its effect varies across benchmarks.

4 Conclusion

The conclusion presents automated generation followed by Difficulty Hiking as a way to obtain useful mathematical reasoning data without expert curation of every problem. The augmentation and hiking ablations support usefulness within the tested setting. Extension to other scientific domains is proposed but not experimentally evaluated.

Limitations

The authors identify dependence on teacher capability and the small training subsets imposed by their compute budget as limitations. They report a 41.9% combined yield for DeepSeek-R1 and 74.0% for GPT-OSS 120B, describing the latter as producing harder questions almost 5.2X faster. The appendix table documents the yield rates but provides no timing or difficulty measurements for those additional claims.

Appendix

• A Pipeline Yield and Teacher Model Ablation: DeepSeek-R1 has a 74.0% self-consistency rate and 56.6% difficulty-filter rate, giving a reported 41.9% combined yield. GPT-OSS 120B raises these to 94.0%, 78.8%, and 74.0%. The combined metric multiplies these two retention rates rather than measuring final retention through every filter.
• B Detailed Analysis of Difficulty Hiking: Figure 3 compares Base, DH, and DH_w_LF difficulty histograms. Hiking reduces counts around ratings 4.0–5.0 and increases them around 6.0–8.0; the length-filtered variant preserves much of this shift.
• C Deeper Analysis of Reasoning and Model Failures: A model judge evaluates 480 generated solutions on a 1–5 scale. LIMO + SAND-Math (DH) improves logical soundness from 3.03 to 3.20, avoidance of hallucination, meaning unsupported content, from 3.54 to 3.64, clarity from 3.03 to 3.10, and conciseness from 2.06 to 2.18. Table 5 lists reasoning errors at 35.8%, misinterpretation at 11.9%, calculation errors at 5.2%, and hallucination errors at 4.6% under “Percentage of Failures”; these entries do not sum to 100%, leaving the complete breakdown unclear.
• D Data Efficiency and Scaling Analysis: Figure 4 plots AIME24 performance for LIMO alone and augmentation with 500 or 1,500 examples. At 1,500 added examples, DH reaches 60.55% versus 59.09% for Base. This supports an advantage at the tested data budget, while three plotted training sizes do not establish a general scaling law.
• E Sensitivity Analysis for De-duplication Threshold: An o4-mini model judge assesses sampled flagged pairs. Thresholds 0.99, 0.97, and 0.95 flag 7.4%, 26.7%, and 44.5% of data, with respective duplicate precision, the proportion of flagged pairs judged true duplicates, of 52.0%, 38.0%, and 24.0%. The selected threshold has the highest measured precision, although 52.0% still indicates substantial filtering of pairs not judged true duplicates.
• F Justification for Decontamination Model Choice: The retrieval embedding model paraphrase-multilingual-MiniLM-L12-v2 encodes 10k samples in 4.94s, compared with 66.54s for gte-Qwen2-1.5B, a reported 13.5× speed advantage. A model judge assigns their top 30 retrieved pairs average similarity scores of 0.41 and 0.36, respectively. This evaluates highly ranked candidates but does not measure recall, the fraction of all benchmark overlaps retrieved.
• G Detailed Training Steps: The appendix documents full-parameter fine-tuning with LLaMA-Factory on one node containing 8× AMD Instinct MI300X GPUs. It supplies training hyperparameters and environment setup information for reproduction.
• G.1 Hyperparameters: Table 9 specifies learning rate 5.0e-6, cosine learning-rate scheduling, warmup ratio 0.0, 10 training epochs, gradient accumulation steps 1, cutoff length 32,768, Flash Attention implementation fa2, and DeepSpeed strategy ZeRO-3. These settings specify the learning-rate schedule, update frequency, sequence-length limit, attention implementation, and distributed training strategy.
• G.2 Training Setup: The documented environment uses the container image rocm/pytorch-training:v25.6, LLaMA-Factory, and deepspeed==0.16.9. Training data are converted to Alpaca-style JSON, a structured example format used by the training framework, registered in data/dataset_info.json, and selected with dataset set to limo,sand_math for augmentation. The launch configuration is examples/train_full/train_sand_math.yaml.
• H Dataset Characteristics: Consistency, Novelty, and Difficulty: This appendix reports contamination of 0.2%, self-consistency of 85%, difficulty retention of 41%, and overall retention of approximately 35%. These figures are not reconciled with the main-body values of approximately 74% self-consistency, 56.6% difficulty retention, and 8,842 retained problems from 23,437 generated sets.
• I Algorithm for Novelty Filtering: Algorithm 1 retrieves \(K = 20\) web results per question, embeds the question and each snippet, and retains the question when maximum cosine similarity, a comparison of vector directions, is at most \(\tau = 0.85\). Its retrieval count differs from the top-10 specification in Section 2.5.
• J Difficulty Hiking Prompt: The template requests a challenging Olympiad problem with an essential but unnamed central theorem, a supporting concept, at least 2–3 non-obvious intermediate steps, and one final integer answer. It also allows greater abstraction or generalization and requests problem and solution text in <Q> and <S> tags.
• K Difficulty Hiking Examples: The authors list adding a central theorem, adding a supporting concept, increasing abstraction, and adding constraints as transformation strategies. They acknowledge that generated rewrites often apply one strategy in isolation rather than combining every requested element. The illustrated ratings rise from 1.0 or 2.0 to 6.0 or 7.0 against a target of 8.0.
• L Question Generation Prompt: The template asks for one novel National or International Olympiad-level problem with a single non-negative integer answer. It combines randomly selected primary and secondary mathematical branches and requests a step-by-step solution with a boxed final answer.
• M Solution Generation Prompt: A separate, short template asks the model to solve an existing mathematics question step by step and place the final answer in boxed{}. This supplies separately sampled solutions for consistency filtering.
• N Question Difficulty Rating Prompt: The rubric assigns scores from 1.0 to 10.0 in 0.5 increments, with references ranging from elementary competition questions to historically difficult Olympiad problems. The judge examines both problem and solution and returns a topic summary, score, and justification in <S>, <D>, and <R> tags.
• O Question Decontamination Prompt: The template compares a synthetic question against retrieved existing questions and requests a yes/no judgment of identity or paraphrase. Its coverage consequently depends on the preceding retrieval stage.
• P Math Theorems and Concepts Taxonomy: The supplied taxonomy, a structured classification of mathematical ingredients, organizes theorems and supporting tools across Number Theory, Algebra, Combinatorics, Geometry, and Probability. It supplies ingredients for Difficulty Hiking and distinguishes conceptual tools from named theorems.

Brief Thoughts

The fixed-size Difficulty Hiking ablation provides the most direct evidence for the proposed rewriting step: using 1,500 hiked examples improves all four reported benchmarks relative to 1,500 Base examples. The 500-example augmentation comparison also demonstrates usefulness, but its 17.85-point headline compares two synthetic augmentation conditions; the improvement over LIMO alone is 4.39 points on AIME25.

Answer agreement and search-based novelty checks leave possible shared solution errors and undetected overlaps. Conflicting retention statistics, top-10 versus \(K = 20\) novelty settings, absent uncertainty estimates, and evaluation of one fine-tuned model limit reproducibility and generalization of the reported gains. The dataset is linked at https://huggingface.co/datasets/amd/SAND-MATH.