MMFactory: A Universal Solution Search Engine for Vision-Language Tasks | Summary

This article explains the key points of MMFactory: A Universal Solution Search Engine for Vision-Language Tasks.

• 2024-12-24 (arXiv)
• Fan, Wan-Cyuan, Rahman, Tanzila, Sigal, Leonid.
• University of British Columbia, Vector Institute for AI, CIFAR AI Chair
• Paper
• Project Page

Read this article in Korean

Summary

• MMFactory addresses how to combine vision-language models, which process images and text, with specialized visual tools into reusable programs selected according to a user’s accuracy and computation constraints. With GPT-4o and the same vision tools as Visual Sketchpad, a system that adjusts tool use using intermediate visual results, it scores 75.3 versus 70.7 on jigsaw solving and 55.4 versus 42.1 on function correspondence in BLINK, a visual-perception benchmark; these tasks involve assembling image pieces and matching objects by function, respectively, and MMFactory scores lower on several other tasks.
• A Solution Router uses large language models (LLMs), models that generate and reason over language, as agents with proposing and reviewing roles to construct multiple executable Python solutions from a task description and labeled examples. A Metric Router selects an evaluation measure and estimates each solution’s performance and computation cost so users can choose among alternatives.
• Reusing a program across a task reduces repeated planning: reported mean execution time is 9.74 seconds per sample versus 19.96 seconds for Visual Sketchpad, with approximately zero online routing time, the time spent coordinating tools, after solution generation. Generating one solution takes approximately 8 minutes, and accuracy gains vary by benchmark: average accuracy on SeedBench, an image-question benchmark covering perception and spatial understanding, rises from 75.6 to 75.8 relative to GPT-4o.

1. Introduction

The introduction identifies three obstacles in existing visual programming, the construction of programs that call visual tools, and tool-integrated multimodal systems: limited choices of tools for each subtask, insufficient attention to user resource constraints, and programs tailored to individual samples. MMFactory searches across vision, language, and vision-language models to produce a family of programs with measured performance and cost.

• Users provide a task description, a few example input-output pairs, and optional constraints. Each resulting program is intended to serve subsequent instances of that task through a fixed executable code path.
• Figure 1 distinguishes this task-level solution search from selecting one model or repeatedly planning tool use for each input.

The diagram makes the unit of reuse explicit: MMFactory samples task examples and produces several programs that combine models, while the comparison systems either plan for each instance or select one model. Its performance-time plot explains why the output is a set of choices.

2. Related works

The related-work discussion connects multimodal models, which process information across modalities such as text and images, with visual programming, model routing, and multi-agent frameworks. Model routing selects among models to balance performance and cost; MMFactory constructs combinations of models and offers multiple candidate programs.

• The paper compares its approach with systems that plan tool calls separately for each instance and routing methods that select a single model.
• Compared with AutoAgents, a framework for automatically creating agents, MMFactory emphasizes vision-model composition, Python execution, and multiple solutions accompanied by performance and cost estimates.

3. Methodology

The methodology separates program construction from evaluation. The Solution Router builds a candidate pool, and the Metric Router evaluates that pool for selection under user preferences; Figure 2 shows how sampled examples, model definitions, metric definitions, and constraints enter these stages.

3.1. Overview of MMFactory

For a task containing N instances, only n example instances have ground-truth answers, with n much smaller than N. The Solution Router uses these examples to generate programs intended to process the entire task, and the Metric Router evaluates the programs on a sampled subset of m instances.

• Each candidate receives a performance and computation-cost pair; other measurements may also be recorded.
• The implementation uses Autogen, a framework supporting multi-agent communication and Python execution, and advanced multimodal language models as routers.

The two-stage flow separates generating candidate programs from evaluating their performance and time. User preferences guide selection from the evaluated pool, linking the model repository to a deployment choice.

3.2. Inputs Structure for Solution Router

The Solution Router receives model definitions describing functionality, arguments, return values, and example usage; predefined requirements; four reference output examples; and the Python code of previously generated solutions. These in-context examples, references included in the prompt to guide the output, are separate from examples sampled from the user’s target task.

• The task-specific input contains a task definition, labeled examples including images, and optional user constraints. If no previous solution exists, the solution-pool field contains “EMPTY”.
• Figure 3 illustrates constraints such as execution time below 5 seconds per sample or models with fewer than 3B parameters, meaning 3 billion parameters; these are illustrative inputs rather than benchmark-wide experimental settings.

The example shows that task specification includes images and correct answers alongside natural-language instructions. Its optional runtime and parameter-count limits illustrate how constraints enter the prompt, without demonstrating that every generated program satisfies them.

3.3. Multi-agent solution router

A solution proposer analyzes existing programs and feedback, then outlines an approach; a solution engineer converts that approach into executable Python. Their outputs use the identifiers ANALYSIS, THOUGHT, and ACTION for analysis, instructions, and code.

• The committee uses a code debugger and code executor to obtain intermediate results, a requirement checker to assess constraint alignment, a code checker to assess execution results and logic, and a repetition checker to detect duplicate solution logic.
• Team leaders exchange proposals and feedback until all committee members accept or the iteration limit is reached. At termination, the latest solution is preserved if the repetition checker finds it nonredundant; reaching the limit therefore does not guarantee unanimous acceptance.

The agent connections explain how execution evidence reaches reviewers: code is modified and executed, then intermediate and final results inform correctness checks. This is the mechanism examined by the debugger and code-checker ablations.

3.4. Metric Router

The Metric Router uses GPT-4 to select a metric from definitions describing use cases, arguments, outputs, and examples. It also receives task instructions, labeled examples, and candidate predictions, allowing selection to reflect both the task and answer format.

• Users can supply a custom metric and bypass automatic metric selection.
• After selection, the system evaluates candidates on a larger sample and plots performance against computation cost. The supplied PDF does not separately quantify the reliability of automatic metric selection.

4. Experiments

Experiments use BLINK, which contains 14 visual perception tasks and 3,807 multiple-choice questions, and SeedBench, which covers 9 tasks with 14k image-question-answer pairs. BLINK emphasizes perception problems that humans can solve at a glance, while SeedBench includes spatial understanding, small objects, and intricate descriptive prompts.

• For these single-choice tasks, GPT maps open-form predictions to the available choices, and string matching determines accuracy.
• Quantitative comparisons hold the multimodal model fixed to the corresponding baseline and use the same vision models as Visual Sketchpad.

4.1. Quantitative Analysis

Table 1 shows gains on selected BLINK tasks relative to GPT-4o alone: jigsaw accuracy increases from 55.3 to 75.3, visual correspondence, matching points across images, from 75.0 to 85.5, and function correspondence from 39.2 to 55.4. Gains are not universal: reflection decreases from 38.8 to 35.1, and the IQ category, which tests visual reasoning, decreases from 30.0 to 28.7.

• With LLaVA-7B, a vision-language model with 7 billion parameters, spatial accuracy increases from 61.5 to 78.8; with its 13-billion-parameter counterpart, LLaVA-13B, jigsaw accuracy increases from 58.0 to 64.0. These comparisons show gains with different fixed base models.
• Against GPT-4o with Visual Sketchpad, MMFactory improves jigsaw from 70.7 to 75.3 and function correspondence from 42.1 to 55.4, but scores lower on depth, counting, localization, and similarity. Table 1 combines previously reported Visual Sketchpad results with results collected using its official codebase.

MMFactory with GPT-4o scores higher than Visual Sketchpad on jigsaw and function correspondence, while Visual Sketchpad scores higher on depth and similarity. The annotations distinguish previously reported results from official-codebase runs, a relevant condition when interpreting the comparison.

Table 2 reports a small SeedBench average improvement, from 75.6 for GPT-4o to 75.8 for MMFactory. Interaction improves from 78.6 to 80.6 and reasoning from 81.2 to 84.5, while instance identity, location, counting, and spatial scores decline.

• The results show benefits in selected categories. No confidence intervals or significance tests accompany the reported average difference.

Improved interaction and reasoning coexist with declines in several other categories. The average increase from 75.6 to 75.8 is therefore much smaller than the strongest individual-category gains.

4.2. Qualitative Analysis

Figure 5 illustrates a spatial-relation task constrained to open-source models and less than 10 seconds per sample. Candidate programs vary from prompt enhancement to object detection followed by vision-language reasoning, and their measured accuracy-time positions allow selection under the runtime constraint.

• One candidate extracts object names, detects the objects with Grounding DINO, an object-detection tool, and appends their bounding boxes, coordinate-based descriptions of object locations, to the prompt before querying LLaVA-7B.
• Intermediate outputs expose extracted names, detected boxes, the augmented prompt, and the final answer. The example demonstrates program inspectability but does not establish a general success rate for constraint satisfaction.

The spatial-relation example connects generated code, intermediate outputs, and selection under a 10-second runtime limit. Different candidates alter prompting, object detection, or the vision-language model, illustrating how programs in the pool differ.

4.3. Model Analysis

The committee ablation, which removes individual components to measure their contribution, uses the first five BLINK tasks in Table 1, three runs per task, and at most six conversation iterations. Table 3 reports full-system accuracy of 50.5 and an execution error rate of 0.0; removing the code debugger reduces accuracy to 40.0, while removing the code checker reduces it to 33.3 and raises the error rate to 20.8.

• Removing the requirement checker yields accuracy 48.1, error rate 0.5, and 2.4 solutions on average. Removing the repetition checker yields accuracy 40.5, error rate 17.8, and 2.0 solutions versus 3.0 for the full system.
• In a separate analysis of 10 randomly selected BLINK tasks with at most six iterations, the authors report that the best solutions generally occur around iterations 2–4. Figure 6 also shows that further refinement can reduce accuracy.

Removing the code checker produces the largest listed accuracy decline and an execution error rate of 20.8, compared with 0.0 for the full system. Removing the repetition checker reduces the average solution count from 3.0 to 2.0, supporting its role in maintaining candidate diversity.

The trajectories show that later conversation steps can undo earlier gains. The authors report that the best solutions generally appear around iterations 2–4, suggesting that additional discussion does not reliably improve accuracy.

Solution generation takes approximately 8 minutes on average, and the measured time per solution increases as the pool grows. Figure 7 reports mean execution times of 9.74 seconds for MMFactory and 19.96 seconds for Visual Sketchpad, with routing times of approximately 0.00 and 18.20 seconds per sample, respectively; the near-zero figure concerns online coordination after reusable programs have been generated.

• Table 4 reports application programming interface (API) calling costs per 10 samples in USD, United States dollars. MMFactory versus Visual Sketchpad costs are 0.064 versus 0.211 for depth, 0.045 versus 0.232 for spatial, 0.041 versus 0.224 for jigsaw, 0.034 versus 0.281 for visual correspondence, and 0.058 versus 0.230 for semantic correspondence, matching image content by meaning.
• These measurements show lower costs in the evaluated comparison. Assessing the benefit of reuse also requires accounting for the separate upfront generation time; the paper does not report a break-even workload.

The upper plot shows the upfront search time and its increase as more solutions are generated. The lower comparison reports faster per-sample execution and approximately zero online routing for MMFactory; assessing total time savings requires considering both generation and reuse.

MMFactory has lower listed API calling costs on all five evaluated tasks, although the reduction varies by task. The values support cost savings in this comparison without establishing a universal reduction factor.

5. Conclusion

The conclusion presents MMFactory as a way to select reusable task solutions from a pool according to user constraints. The reported experiments show improved performance on several perception and reasoning tasks and reduced repeated routing costs, with accuracy advantages depending on the task and comparison method.

Brief Thoughts

Separating program search from repeated task execution is a useful engineering choice: fixed programs expose tool combinations for inspection and reduce recurring coordination. The timing comparison and committee ablations support this design, although the approximately 8-minute generation time means its practical benefit depends on subsequent reuse.

The small SeedBench average difference and several BLINK regressions limit claims of general accuracy improvement. The committee ablation covers only five tasks, and the supplied PDF offers no separate evaluation of metric selection, leaving broader task coverage and evaluation reliability unresolved.

The main text refers to supplementary prompts, agent responses, additional examples, and metric-routing discussion, but those materials are unavailable in the supplied PDF.