optimize_anything: A Universal API for Optimizing any Text Parameter | Summary

This article explains the key points of optimize_anything: A Universal API for Optimizing any Text Parameter.

• 2026-05-19 (arXiv), ACM Conference on AI and Agentic Systems (CAIS ’26)
• Agrawal, Lakshya A, Lee, Donghyun, Tan, Shangyin, Ma, Wenjie, Elmaaroufi, Karim, Sandadi, Rohit, Seshia, Sanjit A., Sen, Koushik, Klein, Dan, Stoica, Ion, et al.
• UC Berkeley, MIT
• Paper
• Project Page

Read this article in Korean

Summary

• Optimizing code, prompts, and agent systems usually requires separate search tools; this paper asks whether one large language model (LLM) system can improve them through a shared text interface. Using Gemini 3 Flash for both search proposals and agent execution, optimize_anything raises accuracy on ARC-AGI, a benchmark of grid-transformation puzzles, from 32.5% for a single-call seed agent to 89.5% for the evolved agent.
• The application programming interface (API) combines an initial artifact, an evaluator, and optional datasets; the evaluator returns a score and optional Side Information (SI), diagnostic feedback describing candidate behavior. Its default backend extends GEPA, a reflective evolutionary search method, with single-task search, shared search across related tasks, and optimization for unseen inputs. A Pareto frontier, a candidate pool preserving complementary strengths across examples or metrics, supports selection and transfer.
• Reported results include 40.2% CloudCast cost savings over Dijkstra shortest-path routing, 87% of generated CUDA kernels, GPU programs implementing individual operations, matching or beating PyTorch, and a circle-packing score of 2.63598 in 63 evaluations versus OpenEvolve’s 2.6307 in 200 evaluations with the same GPT-5.1 proposer. Ablations support diagnostic feedback and transfer between related tasks, but joint search across different circle counts degrades circle-packing results. Performance depends on proposer capability, evaluator expense, and domain expertise in designing feedback.

1 Introduction

The paper identifies a common structure across optimization tasks: serialize an artifact as text, evaluate it automatically, and ask an LLM to revise it using diagnostic feedback. Its declarative interface lets users specify the problem while the system handles prompt construction, reflection, candidate selection, and search. The contribution combines broad artifact support, three optimization modes, and a uniform diagnostic-feedback contract.

The evaluator produces two signals: a scalar for ranking candidates and diagnostics for guiding revisions. The examples share this search interface while retaining domain-specific artifacts and evaluators.

2 Related Work

The comparison covers program-evolution systems AlphaEvolve, OpenEvolve, and ShinkaEvolve; prompt optimizers GEPA, MIPROv2, and TextGrad; iterative self-correction methods Reflexion and Self-Refine; and agent-architecture search systems ADAS and AFlow. optimize_anything uses GEPA’s reflective, Pareto-based search as its default backend and extends it beyond prompts. Table 2 presents the authors’ feature comparison, emphasizing multi-task search and diagnostic feedback exposed through a common interface.

3 The optimize_anything API

The optimize_anything API separates the optimization problem from the search implementation: users specify the artifact, evaluator, optional examples, and domain context. The same interface accepts prompts, executable code, complete agent systems, and code that produces images or models.

3.1 Core Interface

The minimal call supplies seed_candidate and evaluator; the evaluator consumes a candidate string and returns a higher-is-better score with an optional diagnostic dictionary. Additional arguments are dataset, valset, objective, background, and config; oa.Image supplies image feedback to a vision-capable language model (VLM). With seed_candidate=None, seedless mode uses a natural-language objective to generate the initial artifact while retaining the requirement for an evaluator.

3.2 Three Optimization Modes

Single-task search provides no dataset and optimizes one directly evaluated solution, such as a circle-packing algorithm. Multi-task search supplies related problems and shares a frontier during search, then selects a specialized output for each task. Generalization supplies both dataset and valset, a separate validation set, and produces one artifact intended to work on unseen examples.

4 Method

The interface supports multiple optimization backends; the default implementation extends GEPA with frontier types and selection rules for the additional modes. Supporting arbitrary artifacts also requires a refiner that repairs common generation errors before evaluation, caching that reuses results for identical candidate content, typed diagnostic feedback, and adapters between backends and the interface. These are extensions to an existing search method alongside the unified API.

4.1 Problem Formulation

An evaluator maps a text artifact and an optional example to a scalar score and diagnostic information: f(x,e)=(s(x,e),ι(x,e)). Single-task search maximizes the artifact’s direct score; multi-task search maximizes average performance across related problems; generalization uses training feedback while measuring performance on a separate validation set. The formalization gives an aggregate multi-task objective, while Section 3.2 additionally specifies selecting a best specialized candidate for each task.

4.2 Side Information (SI)

SI can contain text such as compiler errors, structured data such as per-test outcomes and objective sub-scores, or images such as rendered candidates. During reflection, the proposer uses this information to diagnose failures and propose targeted revisions; the paper’s comparison with a numerical gradient is an analogy to directional guidance. Feedback is optional, and capture_stdio=True can collect existing printed diagnostics.

4.3 Pareto-Based Search

The search tracks scores separately by example or metric, allowing candidates with specialized strengths to survive despite lower averages. The default backend samples candidates in proportion to the number of objectives on which they achieve the best score, reflects on minibatches of 2–3 examples, and fully evaluates revisions that improve on the minibatch. Preserving these candidates supplies different strategies for subsequent proposals and enables transfer across related tasks.

5 Experiments

The experiments describe artifacts, evaluators, feedback, and outcomes for coding-agent skills, cloud scheduling, ARC-AGI agents, mathematical prompts, CUDA kernels, circle packing, and image generation. Although the paper repeatedly states six primary domains, it presents seven application subsections; mathematical blackbox optimization and seedless 3D modeling receive additional appendix treatment. Proposer models and evaluation budgets vary by domain, limiting direct comparisons across applications.

Each row pairs a domain with its search mode and proposer, showing that the headline results use different models. Preliminary mathematical optimization and 3D modeling also appear, despite the caption’s description of six domains.

The authors’ checkmarks compare artifact types, search modes, feedback, and interface requirements. This feature inventory does not measure optimizer performance under matched budgets.

5.1 Coding Agent Skills (Generalization)

For the Bleve search library, the optimized artifact is a set of skills, natural-language instructions and codebase practices, evaluated by whether a coding agent resolves unseen repository tasks. Feedback includes task descriptions, tool and edit traces, errors, test results, and resolution time. Section 5.1 reports Haiku 4.5 improving from 79.3% to 98.3% and Sonnet 4.5 from 94.8% to 100%, with skills transferring without reoptimization; Figure 2 instead labels Haiku’s learned-skills result as 100.0%.

Haiku’s plotted pass rate rises from 79.3% to 100.0% as duration falls from 173s to 130s; Sonnet rises from 94.8% to 100.0% as duration falls from 285s to 150s. The Haiku endpoint conflicts with Section 5.1’s 98.3%, and the approximately 47% duration reduction applies to Sonnet.

5.2 Cloud Scheduling Algorithms (Generalization)

CloudCast optimizes multi-cloud broadcast routing, while Can’t Be Late chooses between cheaper interruptible SPOT instances and reliable ON_DEMAND instances under deadlines; both use training/validation splits over infrastructure scenarios. CloudCast achieves 40.2% cost savings over Dijkstra routing after evolving provider-aware Steiner tree routing, a broadcast-tree strategy accounting for provider costs. Can’t Be Late achieves 7.8% savings after adding availability tracking, switching-cost analysis, and deadline-dependent thresholds; the paper reports an aggregate score of 96.6 on ADRS, its cloud-infrastructure benchmark, versus 92.9 for OpenEvolve and 72.0 for ShinkaEvolve.

The plots separate validation progress from final test markers and show different evaluation budgets for the two scheduling problems. Final test savings are 40.2% for CloudCast and 7.8% for Can’t Be Late.

5.3 ARC-AGI Agent Architecture (Generalization)

The entire ARC-AGI agent, including code, control flow, helper functions, sub-agent structure, and prompts, is optimized as one text artifact using grid examples, per-puzzle outcomes, execution errors, model outputs, and costs as feedback. With Gemini 3 Flash as both proposer and executing model, a 10-line single-call seed becomes a 300+ line system, raising test accuracy from 32.5% to 89.5%; validation accuracy reaches 93.5%. The resulting pipeline infers rules, generates and verifies code with exec(), permits up to 2 debugging attempts, and falls back to direct model prediction.

Agent improvements occur in discrete jumps, reaching 93.5% validation accuracy. Separate test markers show 32.5% for the seed and 89.5% for the optimized agent.

5.4 AIME Prompt Optimization (Generalization)

The system optimizes GPT-4.1-mini’s system prompt on AIME, the American Invitational Mathematics Examination, training on 2022–2024 problems and testing on AIME 2025. Feedback supplies the problem, reasoning chain, extracted answer, reference answer, and correctness flag. Test accuracy rises from 46.67% to 60.00%, compared with 51.33% reported for MIPROv2 on the same benchmark; validation reaches 57.78%, and the prompt develops into six structured reasoning rules.

The final 57.78% validation score and 60.00% test result come from different evaluation sets. Both exceed the 46.67% baseline, and the plotted validation trajectory contains long plateaus between improvements.

5.5 CUDA Kernel Generation (Multi-Task Search)

Kernel generation covers 31 reference PyTorch operations from KernelBench, a benchmark of GPU operations, evaluated on an NVIDIA V100 32GB GPU with GPT-5 as proposer. The system evolves a kernel-generation prompt across tasks; feedback includes compiler errors, numerical correctness failures, relevant documentation, and measured runtime relative to PyTorch. It reports 87% of kernels matching or beating baseline performance, 48% achieving at least 10% speedup, and 25% achieving at least 20% speedup on the selected operations and hardware.

The correctness curve reaches all tasks, while fewer kernels meet progressively stricter speed thresholds. The text reports 87% matching baseline, 48% reaching 1.1× speedup, and 25% reaching 1.2×; the plotted endpoints appear slightly higher for the latter two thresholds.

5.6 Circle Packing (Single-Task Search)

Circle packing maximizes the sum of radii of n=26 non-overlapping circles inside a unit square, with code evaluated using geometry diagnostics and rendered layouts. The discovered solver combines linear programming (LP), which optimizes radii under linear constraints for fixed centers, with L-BFGS-B, a bounded numerical method for center optimization, and CMA-ES, an evolutionary numerical search method for exploration. In the controlled GPT-5.1 comparison, optimize_anything reaches 2.63598 after 63 evaluations at approximately $3.18; OpenEvolve reaches 2.4583 after 100 evaluations at $1.98 and 2.6307 after 200 at $6.85, while the AlphaEvolve comparison uses its published result.

With GPT-5.1 held fixed, optimize_anything attains 2.63598 after 63 evaluations for $3.18, exceeding OpenEvolve’s 2.6307 after 200 evaluations for $6.85. OpenEvolve’s 100-evaluation run costs $1.98 but reaches only 2.4583; the optimize_anything evaluation entry is displayed as 63 / 100.

The full trajectory shows progress from weak initial solutions, while the magnified panel exposes small differences among the highest scores. AlphaEvolve and ShinkaEvolve comparisons use reported results; Table 3 separately documents the controlled OpenEvolve rerun.

5.7 Image Generation (Multi-Task Search)

Image generation optimizes Scalable Vector Graphics (SVG) code and computer-aided design (CAD) models produced with build123d across four visual goals. A VLM scores one visual aspect per evaluator call on a 0–100 scale, treating aspect scores as separate search objectives. Five human evaluators unanimously prefer the optimized outputs across the four goals, and the pelican-riding-a-bicycle example receives a reported VLM score of 0.726 versus 0.330 for the zero-shot baseline, an output generated without iterative optimization; the paper does not explain the conversion between these scores and the stated 0–100 scale.

5.8 Ablation: Multi-Task vs. Single-Task Search

The main transfer ablation reruns the 10 best multi-task KernelBench problems independently with an equivalent per-problem budget, reporting faster convergence and better outcomes from shared search. Because this subset was selected for strong multi-task performance, the randomly sampled 20-problem appendix comparison provides additional evidence: baseline matching rises from 50.0% to 90.0%, and reaching 1.1× speedup rises from 25.0% to 40.0%. The 1.2× threshold remains 15.0% in both conditions, so final gains do not extend to every threshold.

Shared search performs better than independent search across the displayed thresholds with equivalent per-problem budgets. The comparison uses the 10 best multi-task problems, so its selection condition matters when interpreting transfer gains.

5.9 Ablation: Side Information

On Facility Support Analysis prompt optimization, returning aspect sub-scores alongside the aggregate score reaches a validation score of 0.80 within 100 rollouts, or candidate evaluation runs, compared with approximately 600 for aggregate-only feedback; final test scores are 86.32 and 82.5. Additional ablations report circle-packing performance of 100% versus 93.96% of the best score and single-task KernelBench fractions reaching 1.1× speedup of 32.3% versus 12.9%. In multi-task KernelBench, the corresponding fractions are 40% versus 0%, supporting feedback’s usefulness in these settings without establishing one convergence multiplier across domains.

Aspect sub-scores improve both validation progress and the final test metric. The reported comparison is approximately 100 versus 600 rollouts to reach 0.80 validation score and 86.32 versus 82.5 on the test metric.

The ablation extends feedback evidence to packing and kernels, including single-task mean speedups of 4.11× versus 1.15× and multi-task means of 1.15× versus 1.03×. Rows measure different quantities, so their gains cannot be reduced to one common improvement factor.

5.10 Proposer Sensitivity and Optimization Cost

The proposer comparison shows lower cost and lower final quality with GPT-5-nano: circle packing reaches 2.512 for $0.50 versus GPT-5.1’s 2.636 for $6.00, and AIME reaches 50.0% for $3.71 versus 60.0% for $6.44. Reported experiment costs range from $1 for numerical blackbox optimization to $144.70 for ARC-AGI, with KernelBench costing $140 for 31 kernels. Evaluator execution dominates the most expensive settings; for ARC-AGI, $144 is attributed to the agent and $0.70 to reflection.

6 Why the Framework Works: Optimization Trajectory Analysis

Circle-packing trajectory analysis proposes three mechanisms: diagnostic feedback triggers targeted algorithm changes, the code artifact and refiner prompt improve through a shared frontier, and diverse candidate families prevent early commitment to one approach. Examples include moving to LP when radii collapse and to sequential linear programming (SLP), which makes linearized local updates, when center placement becomes limiting. The feedback ablations test SI’s contribution directly, while the explanations involving module interaction and diversity primarily interpret observed trajectories.

7 Discussion

The paper argues that multi-task search helps when related problems share reusable optimization patterns, as CUDA operations do through vectorized memory access and reduction strategies. It reports a counterexample for circle packing across different circle counts: single-task search scores 2.6360, compared with 2.6313 for MT7 and 2.5973 for MT11, multi-task configurations searching 7 and 11 tasks respectively. These results establish a negative-transfer case for the tested configurations; the broader explanation that different circle counts lack transferable structure remains the authors’ interpretation.

8 Limitations

The authors identify five limitations: proposal quality depends on the LLM, evaluation can be expensive, artifacts must have a text representation or proxy, multi-task gains depend on task relatedness, and useful feedback still requires domain expertise. The proposer ablation, cost accounting, and negative circle-packing transfer experiment substantiate several of these constraints. The common interface reduces search-system configuration while leaving evaluator and diagnostic design to the user.

9 Conclusion

The paper concludes that a declarative interface can support competitive optimization across heterogeneous artifacts through three search modes, diagnostic feedback, and Pareto-based candidate preservation. Its backend-independent design allows search implementations to change behind the same user interface. The implementation is released within the GEPA project at https://github.com/gepa-ai/gepa.

Baseline matching improves from 60.0% in single-task search to 90.0% in both multi-task settings. Only MT20 improves the 1.1× fraction beyond 40.0%, reaching 50.0%; all settings finish at 20.0% for 1.2× speedup.

On the random sample, MT20 improves baseline matching from 50.0% to 90.0% and the fraction reaching 1.1× from 25.0% to 40.0%. Both settings reach 15.0% at 1.2×, providing evidence of transfer with a threshold-dependent benefit.

Both proposers improve over their seeds, but GPT-5-nano finishes lower on both tasks. For circle packing, reducing cost from $6.00 to $0.50 accompanies a score decrease from 2.636 to 2.512.

Reported costs vary substantially by domain, from $1 for numerical blackbox optimization to $144.70 for ARC-AGI. The ARC-AGI breakdown attributes $144 to agent execution and $0.70 to reflection, illustrating how evaluator expense can dominate.

The four goals use between 4 and 13 visual aspects and different model assignments. The table documents the evaluation design and shows that the visual evidence covers a small set of goals.

The paired outputs show changes in scene composition, object detail, and the unicorn’s anatomy after optimization. They illustrate the four evaluated goals and complement the reported human preferences, with no broader visual-quality distribution tested here.

Appendix

• A Use of Generative AI: The authors disclose using ChatGPT, Gemini, Claude, and Cursor to help generate text, tables, graphs, and code. They state that the experimental design and details were their original ideas.
• B Blackbox Mathematical Optimization: On the 56-problem EvalSet benchmark with 8,000 evaluations per problem, optimizing solver code ties Optuna, a numerical optimization framework, on 40 problems, wins on 7, and loses on 9. At a lower budget of 2,000 evaluations on 10 selected problems where Optuna struggles, it wins on 7; this selected-subset result does not establish overall superiority across EvalSet.
• C Seedless Mode: 3D Unicorn: With seed_candidate=None, the system generates a Python script using build123d for modeling and pyrender for rendering from a natural-language objective. Rendered multi-view images supply scoring and feedback, allowing iterative refinement of geometry, proportions, and anatomical detail without an initial code artifact.
• D Detailed Algorithm: Algorithm 1 initializes and evaluates a seed, selects a candidate from the frontier, collects minibatch feedback, proposes a revision, and fully evaluates revisions that improve on that minibatch. It updates candidate scores, prunes dominated candidates, and repeats until the evaluation budget is exhausted; the displayed return rule selects the highest-average candidate. For single-task search, a singleton dataset reduces per-example tracking to per-metric tracking.
• E Multi-Task Scaling Tables: On 10 KernelBench problems, Table 6 reports baseline-matching fractions of 60.0%, 90.0%, and 90.0% for single-task, MT10, and MT20 search, where MT10 and MT20 search 10 and 20 tasks jointly. Fractions reaching 1.1× speedup are 40.0%, 40.0%, and 50.0%, while all reach 20.0% at 1.2×. On 20 randomly sampled problems, Table 7 reports single-task versus MT20 results of 50.0% versus 90.0% at 1.0×, 25.0% versus 40.0% at 1.1×, and 15.0% versus 15.0% at 1.2×.
• F Optimization Trajectory Analysis: Full Details: The appendix traces feedback-driven switches between algorithm families, alternating improvements to code and refiner modules, and the survival of structurally different candidates on separate quality metrics. The refiner achieves a score of 1.93 while the code module scores 0.98; the code then absorbs the LP approach and reaches approximately 2.61, followed by SLP refinements reaching approximately 2.63. These observations describe a search trajectory without independently isolating each proposed mechanism.
• G Proposer Sensitivity and Optimization Cost: Tables 8–9 provide the GPT-5.1 versus GPT-5-nano quality/cost comparison and experiment-level spending. Alongside the main reported totals, SVG optimization costs $18, coding-agent skills $50, and cloud scheduling $52.42; the AIME total of $6.44 comprises $2.17 for reflection and $4.27 for model execution.
• H Image Generation Details: The four goals use 12 aspects for a pelican riding a bicycle, 4 for a 3D unicorn, and 13 each for an octopus on a pipe organ and a sloth steering an excavator. Table 10 identifies Gemini Flash 3.0 for the pelican and Claude Opus 4.6 for the other goals. CAD evaluation uses 3 equally spaced viewpoints, and each evaluator call rates one aspect.
• I Optimized AIME Prompt: The disclosed six-rule prompt covers restating the problem, defining notation and constraints, justifying reasoning, recovering from dead ends, keeping reasoning focused while preserving exactness, and isolating the final answer. It documents the artifact used in the prompt-only experiment.
• J Discovered solutions: The appendix provides excerpts of final artifacts across the evaluated domains. These connect aggregate scores to concrete changes in repository instructions, agent control flow, scheduling logic, GPU implementation, and numerical solver structure. J.1 Coding Agent Skills: Bleve Repository: The optimized SKILL.MD excerpts emphasize diagnosing failures through tests and stack traces, narrowing test scope, making reviewable changes, and adding focused coverage where needed. They illustrate the repository practices encoded in the optimized artifact. J.2 ARC-AGI Agent Architecture: Figure 10 details the generated system’s programmer, validator, code-execution, and fallback components. Validation can trigger up to 2 debugging attempts, and final predictions combine a code-based attempt with a direct-model fallback. J.3 CloudCast Routing Algorithm: The 178-line artifact incorporates provider-aware path weights, candidate selection over cost/time trade-offs, multiple Steiner tree strategies, and greedy allocation that accounts for bandwidth contention. The excerpt shows the search_algorithm entry point, while the supplementary artifact contains the full implementation. J.4 Can’t Be Late Scheduling Policy: The 110-line policy tracks persistent SPOT unavailability, compares switching overhead against expected savings, and changes decisions according to remaining deadline slack. Its _step method illustrates these behaviors, with reset() and additional guards included in the full artifact. J.5 CUDA Kernel: LayerNorm: The highlighted layer-normalization kernel achieves a reported 3.32× speedup over PyTorch. It uses float4 vectorization to load four values at once, separates statistics computation from normalization, and uses __shfl_down_sync for register-level partial-sum exchange within a warp, a group of GPU threads. J.6 Circle Packing Algorithm: The 480+ line artifact combines LP-derived gradients for center optimization, bounded numerical optimization, local linearized updates, evolutionary exploration with restarts, and relocation of small circles. Six seeding strategies, hexagonal, uniform, edge-ring, farthest-point, corner-spokes, and edge-biased hex, provide different starting configurations for n=26. Section 7 separately describes a 900+ line circle-packing artifact, so the paper’s length descriptions are inconsistent.
• K Demonstration: A 4-minute video and accompanying artifacts illustrate ARC-AGI agent evolution and circle-packing optimization through Jupyter notebooks. The demonstrations expose intermediate candidates and diagnostic feedback and are linked at https://drive.google.com/drive/folders/1mfd8xny_YRri5UYwTxKoBs3CJ_cpxpMr.
• L Artifact Availability: The reproduction repository is https://github.com/gepa-ai/optimize-anything-artifact, with materials under acm_cais_artifact_evaluation/ and domain-specific code, README.md files, and saved GEPAState search checkpoints under domains/. Most domains require a CPU host and model API access; KernelBench requires an NVIDIA V100 32GB GPU with CUDA 12.1+. The PDF displays ACM badges for artifact availability, functionality, and reproduced results.

Brief Thoughts

The clearest engineering contribution is the evaluator contract, which exposes diagnostic feedback through one interface across artifact types. The matched-proposer OpenEvolve comparison and SI ablations provide specific evidence for effectiveness, while the selected-task transfer ablation, four-goal visual evaluation, and conflicting Bleve results limit broader claims. Practical use still requires reliable evaluators, useful diagnostics, and evidence that jointly searched tasks share transferable patterns.