Self-Steering Language Models | Summary

Contents

Summary
1 Introduction
2 Related Work
3 DISCIPL
4 Experiments
5 Results and Analysis
6 Limitations, Future Work, and Conclusions
Author Contributions
Appendix
Brief Thoughts

This article explains the key points of Self-Steering Language Models.

2025-04-09 (arXiv), COLM 2025
Grand, Gabriel, Tenenbaum, Joshua B., Mansinghka, Vikash K., Lew, Alexander K., Andreas, Jacob.
Massachusetts Institute of Technology, Yale University
Paper

Summary

Language models struggle to generate fluent text while satisfying exact constraints, such as character counts, word positions, and budgets. On COLLIE sentence tasks, a constrained text generation benchmark, DISCIPL (Distributional Constraints by Inference Programming with Language Models) uses a GPT-4o-generated program and a Llama-3.2-1B execution model with importance sampling, which weights independently generated candidates, to reach 0.84 Pass@1—the expected probability that one selected answer satisfies the constraints—at a budget of 32 candidates, compared with 0.07 for direct Llama sampling, 0.46 for GPT-4o with stepwise verbal reasoning, and 0.96 for o1.
DISCIPL assigns a Planner model to write a task-specific inference program, an executable procedure that coordinates generation and scoring by Follower models. It requires no finetuning and uses LLAMPPL, a Python framework for probabilistic programming that expresses probabilistic models and inference procedures as code.
Sequential Monte Carlo (SMC), which repeatedly replaces low-scoring partial generations with copies of promising ones, improves coherency, the paper’s measure of fluency, relative to importance sampling (IS). With the 1B Follower, SMC obtains sentence, paragraph, and PUZZLES—a custom dataset of four constrained writing tasks—Pass@1 scores of 0.76, 0.69, and 0.42 at respective budgets of 32, 8, and 16 candidates, while direct sampling obtains 0.07, 0.38, and 0.08.
Generated programs can contain bugs, and the validity gains from SMC and IS accompany lower fluency scores than direct generation; o1 remains better on aggregate validity in all three domains. The sentence-domain cost analysis reports $4.2606 per 100 tasks for DISCIPL-SMC versus $4.7457 for o1 under the paper’s pricing and prompt-caching assumptions, with the long Planner prompt dominating DISCIPL’s cost.

1 Introduction

The introduction identifies a mismatch between language models’ ability to describe a problem’s structure and their ability to execute exact counting or constrained generation. Chain-of-thought (CoT), reasoning expressed as successive natural-language steps, can be expensive and unreliable, while externally designed search procedures require task-specific engineering. DISCIPL delegates the specification of generation, verification, and search to a model-generated program.

The paper connects inference-time search, parallel generation, automated system improvement, constrained decoding, SMC, and probabilistic programming. Its contribution is to have a language model generate executable inference procedures with control over individual generated tokens and their probabilities, rather than requiring a manually specified task scorer or fixed search procedure.

Prior SMC approaches already steer language models, but adapting their generation and scoring mechanisms to new tasks requires engineering or learning; DISCIPL asks the Planner to write these mechanisms.
Python supports more general computation than specialized markup for parallel generation, although this flexibility also permits program errors.

3 DISCIPL

DISCIPL separates inference design from execution. A Planner writes an inference program, and a general inference engine executes it using a Follower and a specified sampling budget.

3.1 General Framework

Both models must support efficient conditional generation, meaning generation given preceding context; the Follower must also efficiently evaluate conditional probabilities. A generated program can make asynchronous Follower calls, allowing multiple requests to proceed concurrently, to sample text or evaluate its likelihood. The inference engine returns either an answer or a runtime error.

Algorithm 1 prompts the Planner with the task, executes the resulting program, and returns its output when execution succeeds.
On a runtime failure, the Planner receives the error and traceback, a record of the failing execution, and attempts a correction within the retry limit. This feedback does not automatically detect logically incorrect programs that execute successfully.

3.2 Our Instantiation: Language Model Probabilistic Programs

LLAMPPL programs repeatedly extend partial generations and multiply their scores by task-dependent updates. A zero update can encode a hard constraint violation; other updates can express preferences through symbolic rules, model judgments, or Follower probabilities. Forcing a required word with observe() also scores how probable that word is in its context.

The induced target distribution, the probability distribution the inference engine aims to sample, weights each program execution by the product of its score updates.
The program specifies how candidates are generated and scored; whether this correctly implements the task depends on the generated code.

The experiments compare three inference methods: IS generates full candidates independently and samples an output according to their accumulated weights; SMC additionally resamples partial candidates during generation; rejection sampling (RS) generates full candidates and checks their validity afterward. IS converges to the program-defined target distribution when that distribution is well-defined, and SMC does so under the paper’s stated mild technical conditions as the budget grows without bound.

SMC maintains particles, candidate generations together with their execution states, and reallocates computation toward promising partial outputs.
These convergence statements concern the distribution defined by the program; they do not establish that the program correctly represents the task or guarantee success at a finite budget.

3.3 Common Patterns

The Planner chooses the size of each extend-and-score step, such as a word, a poem line, or a span ending at the next constraint. Separate proposal and prior prompts let generation follow task instructions while probability-based scoring favors fluent text: the proposal generates candidates, and the prior supplies reference probabilities used to score them.

Token masks restrict which tokens can be generated, with importance weights, multiplicative corrections for biased sampling, accounting for the distributional distortion introduced by those restrictions.
Self-hinting injects computed state, such as a remaining budget, into the Follower’s context through hint().
Self-checking uses a generated check() method, optionally with external libraries, to verify completed outputs.

Token masking, self-hinting, and self-checking patterns for inference programs.

4 Experiments

The experiments evaluate programs generated for task types excluded from the Planner’s examples. They measure validity and coherency across inference budgets and Follower models, comparing DISCIPL with direct generation, stochastic beam search, which maintains multiple candidate continuations during decoding, CoT prompting, o1, and an expert-reference condition.

4.1 Domains

COLLIE-v1 provides text constraints composed from a formal grammar; the evaluation uses its sentence and paragraph tasks. PUZZLES covers structured poetry, a constrained grant abstract, an ingredients list with a monetary budget, and a multi-day itinerary.

The COLLIE evaluation covers 9 of 13 task types using instances from its Wikipedia corpus.
Word-level tasks are excluded because model token boundaries complicate fine-grained constraints and recovering character-level distributions is expensive.

4.2 Evaluation Metrics

Validity is measured by expected Pass@1 using inference weights where available and uniform weights for baselines. This evaluates the chance that the selected answer is valid without assuming access to an oracle verifier, a perfect external checker that can select a correct answer from multiple candidates.

Coherency is scored from 1 to 10 by GPT-4o-mini using a zero-shot judging prompt, which supplies instructions without scored examples.
Coherency includes invalid generations, so a method can receive a high fluency score while frequently violating task constraints.

4.3 Experiment Setup

The Planner is gpt-4o-2024-08-06 and the default Follower is Llama-3.2-1B-Instruct. The Planner receives a README.md-style tutorial and manually written example programs, with the example for the target task type held out; PUZZLES prompts additionally include COLLIE examples.

The Follower-only baseline samples N independent completions, where N is the inference budget, with temperature=1.0; stochastic beam search uses N as its beam size.
Planner-only baselines include GPT-4o directly and with CoT, plus gpt-4o-mini-2024-07-18.
The o1-2024-12-17 baseline uses max_tokens=25000 and is queried once per task instance because of cost; it is a performance reference rather than a comparison at matched compute. DISCIPL* gives the Planner the target task’s reference implementation.

5 Results and Analysis

Table 1 reports aggregate validity and coherency across the three domains. SMC and IS substantially improve validity over direct 1B Follower generation while lowering coherency scores; their performance relative to direct Planner generation depends on the domain.

5.1 Validity

On COLLIE sentences with N=32, DISCIPL-IS reaches 0.84 Pass@1 and DISCIPL-SMC reaches 0.76, compared with 0.07 for direct Follower sampling, 0.08 for beam search, 0.32 for GPT-4o, 0.46 for GPT-4o with CoT, and 0.96 for o1. These results show that executing a Planner-written procedure can outperform asking the Planner to answer directly in this domain.

Generated check() methods generally agree closely with ground-truth verifiers, but checking alone is insufficient: DISCIPL-RS reaches only 0.20 sentence Pass@1.
The sentence expert-reference variants reach 0.90 with SMC and 0.91 with IS, leaving a measurable gap from automatically generated programs.

COLLIE sentence validity at a fixed budget and across increasing sample or particle budgets.

For COLLIE paragraphs with N=8, SMC improves Pass@1 from the Follower’s 0.38 to 0.69, while GPT-4o achieves 0.80 and o1 achieves 0.95. For PUZZLES with N=16, SMC reaches 0.42 versus 0.08 for the Follower, 0.25 for GPT-4o, 0.30 for GPT-4o with CoT, and 0.82 for o1.

PUZZLES performance varies by task: SMC scores 0.20 on grant_proposal, 0.50 on ingredients_list, 0.70 on square_poem, and 0.30 on trip_itinerary.
The PUZZLES expert-reference SMC score is 0.70, indicating substantial room to improve generated inference programs.

Aggregate Pass@1 and coherency results for COLLIE sentences, COLLIE paragraphs, and PUZZLES.

5.2 Coherency

SMC achieves higher aggregate coherency than IS in every evaluated domain: 5.61 versus 5.07 for sentences, 6.72 versus 5.89 for paragraphs, and 6.40 versus 5.65 for PUZZLES. Figure 4 shows sentence coherency improving with particle count, consistent with intermediate resampling removing awkward partial generations.

Direct Follower generation scores 8.12, 8.28, and 8.60 in the same domains, despite much lower validity.
On sent_04, SMC Pass@1 decreases for N > 8. The authors attribute this to word-level resampling eliminating required words with low prior probability, showing that increasing the budget can worsen validity under a poorly aligned decomposition.

Coherency scaling and score distributions separated by output validity.

Agreement between automatically generated validity checks and ground-truth verifiers across domains.

5.3 Scaling the Follower Model

With the Planner fixed, sentence SMC Pass@1 rises from 0.76 with Llama-3.2-1B to 0.83 with Llama-3.2-3B and 0.85 with Llama-3.1-8B; paragraph scores rise from 0.69 to 0.74 and 0.75. Improvements depend on the inference method and task.

The size comparison changes from Llama-3.2 to Llama-3.1 at 8B, limiting attribution to parameter count alone.
Qwen3-1.7B raises sentence SMC Pass@1 to 0.83, a gain of 0.07 over the 1B Llama Follower. The paper reports no significant paragraph difference between these model families.
Coherency does not increase monotonically in every setting: sentence SMC coherency is 5.61, 6.36, and 6.07 across the three Llama sizes.

Effects of Follower parameter count and model family on constrained-generation validity.

6 Limitations, Future Work, and Conclusions

The demonstrated scope is constrained text generation with a larger Planner and smaller Follower. Mathematical reasoning, soft constraints that express preferences rather than mandatory requirements, backtracking search that revisits earlier decisions, iterative editing, and using the same model for both roles are proposed extensions rather than established experimental results.

Within the evaluated domains, model-generated programs coordinate parallel inference and substantially improve small-model constraint satisfaction.
Broader applicability remains untested, and the reported program failures and validity–coherency tradeoff constrain the current results.

Author Contributions

Gabriel Grand led research conception, narrative development, software design and implementation, experiments, analysis, figures, and writing. Joshua B. Tenenbaum and Vikash K. Mansinghka contributed senior mentorship and narrative development. Alexander K. Lew additionally contributed conception, mathematical formalisms, analysis, figures, and writing, while Jacob Andreas contributed conception, analysis, and writing alongside mentorship and narrative development.

Appendix

A SMC Visualization traces N=16 particles through a sentence requiring exactly 18 words, with “Glasgow”, “in”, and “and” at positions 4, 8, and 11. Unlike the simplified word-by-word illustration in Figure 1, its first three steps generate spans through the next required word, and its final step completes the sentence. Resampling favors candidates with higher probability under the constraints.
B COLLIE Dataset and Results provides task definitions and detailed sentence and paragraph results. The four sentence task types contain 38, 98, 29, and 94 instances; the five paragraph task types contain 9, 94, 93, 18, and 89 instances. Aggregate metrics give equal weight to constraint types within each text level rather than allowing larger task groups to dominate.
C Puzzles Dataset and Results reports 10 runs of each method on each of four task types. Conditions include an 8-line poem with exactly 8 words per line, a 75–100-word grant abstract avoiding specified terms, at most 7 ingredients costing less than $18.00, and a 3-day Singapore itinerary with at least four activities and at least 9 hours of free time each day. Tables 5 and 6 provide example valid outputs and task-level results.
D Weighted Pass@1 defines the metric as the fraction of total sample weight assigned to valid outputs: Weighted Pass@1 = sum of weights of passing samples / sum of weights of all samples. Unnormalized weights are obtained by exponentiating log weights; null outputs receive zero weight. The metric is unchanged by a common positive rescaling of weights and reduces to ordinary Pass@1 when all samples have equal weight.
E Coherency Score specifies gpt-4o-mini-2024-07-18 as the judge, replacing the original evaluation’s GPT-4 to reduce cost and inference time. The prompt requests an analysis of at most 3 sentences ending in a score between 1 and 10. The paper does not report a comparison with human ratings.
F Planner-Only Chain-of-Thought and F.1 Chain-of-Thought Prompt document the stepwise-reasoning baseline, whose final answer is enclosed in <answer></answer> tags. F.2 Error Examples includes F.2.1 Character Counting, where GPT-4o claims 82 characters but returns 90, and F.2.2 Word Positioning, where required words miss positions 8 and 11 despite an explicit positional plan.
G DISCIPL Error Analysis reports that traceback feedback corrects roughly half of initial runtime errors, but errors remain after 3 attempts: 2–6% on COLLIE and 18% on PUZZLES. Frequent causes include token masks that exclude every possible next token and timeouts associated with faulty termination logic. Listings 1 and 2 show an extra punctuation character after reaching the target length and a case-normalization bug that prevents capitalized target words from matching.
H Inference Methods formalizes SMC and limits execution to T=1000 steps, with an additional wall-clock timeout for loops inside step(). I Implementation Details uses the vLLM backend, a language-model serving system, and sets baseline max_tokens to 32 for sentences, 128 for paragraphs, and 512 for PUZZLES. DISCIPL uses Planner-selected token limits, R=3 retries, timeouts of 120s, 240s, and 360s, and maximum budgets of N=32, N=8, and N=16 for the respective domains.
J Compute Cost Analysis and Key Takeaways examine COLLIE sentence tasks. Table 7 gives costs per 100 tasks of $0.0004 for Follower-only, $0.0709 for GPT-4o, $0.3612 for GPT-4o CoT, $4.7457 for o1, and $4.2606 for DISCIPL-SMC with N=32. DISCIPL programs use 40.1% fewer intermediate tokens than o1 reasoning traces and cost 80.2% less to generate, but the full-system cost advantage is modest because Planner prompting dominates; Follower generation accounts for <0.1% of DISCIPL’s estimated cost in this setting.
Column Descriptions: distinguishes answer generation, reasoning/program generation, task prompting, and cached or non-cached Planner prompting. Token Counts and API Pricing Details uses prices as of 2025-05-31 and estimates local Follower computation using Together API prices as a conservative upper bound; Average Token Counts and Standard Deviations per Task reports 456.9 (69.4) program tokens for DISCIPL versus 764.8 (1018.3) reasoning tokens for o1. The authors report that 95.6% of Planner prompt tokens were cached, meaning previously processed prompt content was reused at discounted pricing, and that static prompt prefilling, processing prompt tokens before generation, accounts for 72% of DISCIPL’s cost; reducing this overhead is proposed rather than demonstrated.
K Prompts and Code Links provides versioned reproduction resources. K.1 Planner System Prompt links to https://github.com/gabegrand/self-steering/blob/v1.0.0/disciple/prompts/planner_system_prompt.md. K.2 Example Models lists COLLIE Example Models at https://github.com/gabegrand/self-steering/blob/v1.0.0/evaluations/collie_eval/models.py and PUZZLES Example Models at https://github.com/gabegrand/self-steering/blob/v1.0.0/evaluations/puzzle/models.py, with target-task examples omitted in the main leave-one-out evaluation.

Brief Thoughts

The contribution is an executable interface for generating task-specific inference procedures. The sentence results show a substantial advantage over direct generation and rejection sampling, while the gap between automatically generated and expert-reference PUZZLES programs identifies program quality as a practical limitation.

An engineering assessment should consider validity, fluency, and total cost together. The reported constraint-following gains are substantial for small Followers, but lower coherency scores and prompt-dominated costs leave the preferred system dependent on the application’s requirements.