3DCodeVerse: Exploring 3D Worlds as Code | Summary

This article explains the key points of 3DCodeVerse: Exploring 3D Worlds as Code.

• 2026-10-15 (arXiv)
• Gao, Yipeng, Zeng, Ziyao, Huang, Jinfa, Li, Yijiang, Xiong, Haolin, Qin, Wang, Yu, Xinlei, Yao, Youheng, Zhang, Jie, Lin, Kun-Yu, et al.
• University of Southern California, Yale University, University of Rochester, University of California, San Diego, Columbia University, Nanyang Technological University, University of California, Santa Cruz, Google DeepMind, Philo Labs
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Generating editable 3D content as executable programs lacks shared training data and verification infrastructure across coding environments. 3DCodeVerse Harness, a workflow that executes and checks coding agents’ outputs, achieves 90% human preference against Claude Design on 3D objects when both use Claude Opus 5.5.
• The contribution combines 3DCodeVerse-2M with automated planning, code assembly, geometric checks, and visual evaluation. The dataset contains 2,028,286 exported rows across several representations, while its packaged catalogue contains 269,597 unique executable programs; the headline scale therefore counts exported examples rather than distinct programs.
• Fine-tuning raises execution on 3DCodeBench, a benchmark of procedural 3D modeling through code, from 0.0% to 84.9% for Qwen3.5-9B and from 51.9% to 92.9% for Qwen3.8-27B at decoding temperature T=0.7 with one seed. The harness also improves human preference and scores against written quality criteria across the main comparisons, but uses substantially more generation compute than the single-pass baselines, and some appendix comparisons show smaller or negative gains.

1 Introduction

The paper treats a 3D program as an editable construction specification that can encode geometry, articulated motion through connected moving parts, scene composition, and graphics effects. Existing resources provide supervision or feedback within particular environments, but offer limited shared infrastructure across these tasks.

• 3DCodeVerse connects corpus construction, program generation, verification, repair, and model training. Successful programs, failed attempts, repairs, and preference comparisons can become training records.
• Figure 1 illustrates static objects, articulated objects, scenes, and graphics produced entirely through code. Table 1 compares this coverage with prior datasets and harnesses; its reported sizes use different counting units and are not directly comparable.

The examples establish the intended coverage across objects, moving assemblies, scenes, and graphics. Their common representation is executable code, which supports editing and inspection; the showcase does not quantify generation reliability.

The comparison positions 3DCodeVerse as combining broad content coverage, multiple code types, iterative refinement, visual feedback, and geometry checks. Counts across rows use different units, so the table supports a coverage comparison rather than a normalized ranking by dataset size.

2 Related Work

The related work distinguishes supervision over construction procedures from generation workflows that execute, inspect, and revise programs. DeepCAD and Fusion 360 Gallery expose computer-aided design (CAD) construction sequences; other resources provide executable scripts, procedurally generated scenes, or graphics programs.

• SceneCraft, BlenderAlchemy, CADCodeVerify, and VIGA provide precedents for planning or visual refinement. Articraft adds structured validation for articulated assets.
• The contribution integrates reusable execution and verification across coding environments with automated training-data curation. Program generation, rendering feedback, and structural validation each have prior examples.

3 3Dcodeverse-2m: A Large-SCALE 3D Coding DATASET

A task specifies a condition c, consisting of a text request and optional reference images, and a declared execution environment e. Executing a standalone program produces an artifact and execution feedback; success requires both a runnable program and satisfactory 3D content.

• Equation (1), (a, ℓ) = E_e(f), x = I[a ≠ ⊥], separates artifact production from failure: f is the program, a is the artifact, ℓ is execution feedback, ⊥ denotes failure, and I is an indicator of successful artifact production.
• A large language model (LLM), the model generating the program, can produce code directly. A coding agent instead revises code using earlier programs, runtime feedback, and rendered observations, as formalized in Equation (2).

3DCodeVerse-2M combines collected programs, converted representations, and newly generated programs. DeepCAD construction sequences become standalone CadQuery scripts for programmatic CAD modeling, while Articraft geometry routines are incorporated into standalone programs and its Unified Robot Description Format (URDF), which specifies links and joints, is retained.

• Inspection covers part inventories, bounding boxes, support, contact, collision, sampled joint poses, and rendered frames. Conversion fidelity uses voxel intersection over union (IoU), the overlap of occupied volumes, for DeepCAD and Chamfer distance, a distance between sampled shapes, for Articraft.
• A vision-language model (VLM), which processes images alongside text, supplies instruction, visual-description, and construction-procedure captions from code and renders. Records retain provenance, licenses, associated assets, and separate execution, geometric, and visual verification signals.
• Verification coverage varies by source. Collected and converted programs receive execution and rendering checks where runtimes are available; harness-generated programs additionally carry gate results from deterministic checks and visual-judge results.

Collected programs and newly generated code enter a shared build, inspection, and repair process. Meshes and videos accompany code, renders, and captions when available; the surrounding text specifies that verification coverage varies by source.

4 3DCodeVerse HARNESS: A DATA Engine for 3D Coding

The harness translates a request into a structural plan covering components, dimensions, materials, spatial relations, and acceptance criteria. It organizes coding and assembly as a directed acyclic graph (DAG), a dependency graph without cycles, with explicit file ownership for independent agent tasks.

• Each round gathers execution logs, numerical spatial measurements, and visual judgments. Failed execution produces no artifact, leaving execution feedback as the only available feedback.
• Spatial tools return component-level violations and numerical corrections. A visual judge examines multiple rendered views against the request and acceptance criteria, then proposes revisions.
• The score combines weighted quality criteria and defect penalties, subject to upper bounds from measured violations and unmet requirements. Applicable passing geometric checks can veto contradictory visual defect claims; repair tasks prioritize execution, measurements, and then visual feedback.

The development-time adversarial loop alternates between revising the judge and revising the generation workflow. Judge revisions are tested on a fixed bank of artifacts to separate evaluator changes from generation variability, while workflow revisions are compared on matched prompts.

• Equation (3) expresses a consistency objective combining repeated-score variance with claims contradicted by measurements. The paper explicitly states that sensitivity to differences in artifact quality remains an additional requirement.
• Equation (4) balances selected-output scores against observable workflow failures such as invalid plans, redundant tool retries, and failed builds. These objectives describe the criteria for harness engineering; the paper does not report an end-to-end gradient-based optimization of the workflow.

The diagram connects dependency-based code assembly with execution, spatial checks, rendering, judgment, and targeted repair. The displayed round scores illustrate candidate improvement and selection in one trajectory rather than an aggregate experimental result.

5 Experiments

The experiments assess whether the harness improves existing coding agents and whether the corpus improves open models through training. They distinguish execution reliability from reference-shape fidelity, human preference, and automated quality scores.

5.1 RESULTS of 3DCodeVerse HARNESS

The main evaluation describes a pool of 100 text-to-3D object prompts, articulated-object generation, 50 scene prompts, and 50 graphics prompts. Human Preference and VLM Preference measure anonymized pairwise choices with ties counted as half a win; Judge Score is Gemini 3.1 Pro’s score against weighted quality criteria, penalties, and caps on a 0–100 scale, with failed generations scored zero.

• For objects, human preference for the harness ranges from 69% to 89% against the corresponding native workflow. Table 14 gives Judge Score changes of 61.3 to 86.6 for Gemini 3.8 Flash, 77.2 to 88.5 for Claude Opus 5.5, 76.4 to 87.7 for GPT-6 Astra, and 64.4 to 85.9 for GPT-6 Luna.
• Against Claude Design using Claude Opus 5.5, the harness with the same backbone receives 90% human preference and scores 87.7 versus 37.4 in Table 14. Figure 4 rounds these scores to 88 and 37.
• Figure 4 reports human preference of 78–88% on articulated objects, scenes, and graphics. SigLIP-2, a text–image encoder used to measure prompt–render similarity, provides an additional metric averaged over four object views.

Appendix C.3 specifies zero subsequent refinement rounds, T=0, for the primary object experiment, while retaining initial planned generation, interactive tools, automated build repair, and final verification. It evaluates Gemini 3.8 Flash and Claude Opus 5.5 on 60 prompts and several other backbones on a balanced 30-prompt subset; separate studies allow T=4 refinement rounds, using T here as a round count rather than decoding temperature.

• Final evaluation rebuilds delivered programs from scratch and removes internal plans, checklists, and refinement histories. Gemini 3.1 Pro scores each output three times against the raw request and a prompt-derived checklist.
• These safeguards and the human preferences support an advantage for the complete workflow. Generation budgets are unequal: Table 8 estimates $3.312 per object for Flash with the harness versus $0.035 for its baseline, excluding post-run evaluation.
• An additional uncapped visual score excludes the proportions-and-scale criterion and aggregate penalties and caps. On the 60-prompt comparison, Flash with the harness scores 88.1 versus Claude Design’s 62.4, showing that the reported advantage persists under this alternative scoring rule.

Matched-backbone comparisons show higher human preference for the harness across the four displayed tasks. Object Judge Scores also rise, including the same-backbone Claude Design comparison; these are complete-workflow comparisons with unequal generation budgets.

5.2 Fine-Tuning OPEN Models on 3Dcodeverse-2m

The authors fine-tune Qwen3.5-9B and Qwen3.8-27B using low-rank adaptation (LoRA), which trains compact parameter updates, in the LLaMA-Factory training framework. The reported stages include supervised fine-tuning (SFT), learning from request–program examples, and direct preference optimization (DPO), learning from preferred and dispreferred completions.

• Evaluation covers 805 fixed tasks: 212 from 3DCodeBench, 103 Blender objects, 200 CadQuery parts, 50 OpenSCAD models, 200 OpenGL Shading Language (GLSL) shaders, programs that compute rendered appearance, and 40 Three.js single-object scenes. Blender is a 3D creation environment, OpenSCAD provides script-based solid modeling, and Three.js supports browser-based 3D graphics.
• Models receive identical requests and per-dialect system prompts without examples, use decoding temperature T=0.7 with one seed, and execute in runtimes fixed to specified versions. Execution means producing a result in that runtime; for GLSL, it means compiling the shader.
• Shape fidelity uses F@0.05, the surface-point F-score at distance threshold 0.05 after size normalization, and DINOv2-L image-feature similarity between four-view clay renders that suppress material differences. Failed programs receive zero on both metrics.

Table 2 reports execution gains from 0.0% to 84.9% on 3DCodeBench for Qwen3.5-9B and from 51.9% to 92.9% for Qwen3.8-27B. Table 3 also reports higher reference fidelity: 3DCodeBench F@0.05 rises from 0.000 to 0.317 for the 9B model and from 0.214 to 0.374 for the 27B model.

• The 9B model reaches 90.3% execution on Blender and 98.5% on CadQuery; the 27B model reaches 93.2% and 98.5%, respectively. The pretrained 9B model frequently calls scene-clearing operators unavailable in Blender 5, so adaptation to the evaluated runtime contributes to these gains.
• Execution and fidelity remain distinct. For example, Table 3 reports an almost unchanged 27B OpenSCAD F@0.05 of 0.415 to 0.416 despite execution increasing from 80.0% to 90.0%, although the final fidelity value conflicts with the appendix.
• Table 2 reports 74.5% Three.js execution for 9B + SFT, whereas Table 11 lists 32.5% for the 270K mixture matching its other results. Table 3 reports 0.416 OpenSCAD F@0.05 for 27B + SFT + DPO, whereas Table 11 lists 0.314. The PDF does not reconcile these discrepancies.

Training improves execution strongly, especially where pretrained models rarely run in the evaluated environment. The 3DCodeBench changes are 0.0% to 84.9% for 9B and 51.9% to 92.9% for 27B; execution alone does not establish shape fidelity, and the reported 9B Three.js value differs from Table 11.

5.3 Qualitative Analysis

The matched examples in Figure 5 show more complete structures and finer details with the harness, including the phoenix’s legs and nest and the piano’s interior. Successive revisions improve facial feathers, attach stool legs to the seat, and reduce block artifacts in a neon shader.

• These selected examples illustrate repairs without additional human instructions. Their scores are comparable within each matched pair or refinement sequence and do not estimate the average improvement across tasks.

The selected pairs show changes in structural completeness and detail, while refinement sequences localize repairs to feathers, stool attachments, and shader artifacts. Scores are comparable within each pair or sequence, as specified by the paper.

Reference fidelity improves on most reported suites, including 3DCodeBench F@0.05 from 0.214 to 0.374 for 27B. The reported OpenSCAD change is almost flat despite higher execution, but its final value conflicts with Table 11 and remains unresolved.

6 Applications of 3D Coding

The robotics application uses code to specify editable rigid-link geometry, joint axes, and joint limits for the MuJoCo physics simulator. Across 12 cases covering two assets, two initial contact states, and three control sequences, direct code renders and reconstructions through the frozen variational autoencoder (VAE), an image encoder–decoder in Unified World Models (UWM), each preserve the tested door-and-handle geometry in 12/12 outputs; UWM’s action-conditioned video predictions pass in 7/36 rollouts.

• This is a preliminary structural-preservation experiment. Code simulation receives complete scene state, while UWM receives image history and matched robot commands, so the comparison does not establish an advantage in state inference or robot-task success.
• For video generation, rendered code sequences condition Gemini Omni Flash and Seedance 2.5. Programs specify layout, object motion, and camera trajectories, while the video models refine appearance; Figure 6 provides qualitative examples rather than a quantitative video benchmark.

The paired frames illustrate code renders supplying spatial and motion guidance to video models that modify appearance. They demonstrate the proposed application interface; quantitative robotics geometry results come from a separate controlled experiment.

7 Conclusion

The paper presents executable datasets, reusable verification infrastructure, and open-model baselines for generating 3D content through code. It reports improved executability after training and higher content quality with the harness, and states an intention to release the datasets, harness, models, and training recipes.

• The PDF lists https://3dcodeverse.com, https://github.com/gaoypeng/3dcodeverse, and https://huggingface.co/datasets/ilabai/3dcodeverse.

Appendix

• A THE 3DCODEVERSE HARNESS: DESIGN AND IMPLEMENTATION describes the operational system behind generation and curation: modular model roles, structured planning, deterministic verification, visual judgment, revision scheduling, and archived outputs.
• A.1 COMPONENTS AND MODEL ROLES separates the planner, coding agent, and visual judge across four tracks and seven isolated runtimes. These cover Blender Python, CadQuery, and Three.js objects; URDF with Blender-authored links; Three.js scenes; GLSL; and moderngl, a Python interface for OpenGL graphics. Agents access spatial tools through the Model Context Protocol (MCP), the tool interface used by the harness.
• A.2 PLANNING AND THE JOB GRAPH specifies named parts, metric bounds, materials, attachments, and acceptance criteria. For modular objects with at least eight top-level parts, sessions handle attachment subtrees of at most three parts before assembly; refinement tasks sharing files are grouped together. Invalid kinematic plans, which describe links and their permitted motion, trigger a fresh planning context when they reference undeclared links.
• A.3 SPATIAL TOOLS AND DETERMINISTIC GATES separates a valid negative verification verdict from a tool-protocol error, preventing measured defects from triggering transport retries. Connectivity and plan-adherence checks return numerical corrections in the program’s native coordinate frame, and measured violations constrain scores. Figure 9 specifies contact at ≤2 mm and bounding-box tolerance max(1 cm, 10% of planned extent), with an error beyond three times that tolerance.
• A.4 AUTOMATED JUDGMENT requires the visual judge to report observations before assigning scores, without access to source code or generator reasoning. Criterion scores are averaged, defect votes use majority aggregation, and applicable deterministic checks control penalties and score caps; a structural-defect veto requires the corresponding gate to pass without error and without a penetration warning of 8 mm or deeper. Repeated evaluation of fixed chair and cabinet artifacts gives Gemini 3.1 Pro a mean score standard deviation of 0.030 on a [0,1] scale under shuffled montage order, compared with 0.083 for Gemini 3.8 Flash.
• A.5 REFINEMENT ROUNDS, HISTORY, AND OUTPUT SELECTION describes fresh build, measurement, rendering, judgment, and archival for each round, with up to three targeted build-repair sessions. The standard profile permits T=4 refinement rounds and a 60-minute wall-clock ceiling, with early stopping for resolved verification tasks, no code change, or exhausted budget. Output selection ranks archived rounds by composite score, then gate errors and round order; when the top two scores differ by at most 0.03, an optional pairwise tie-breaker promotes the runner-up only if it wins both display orders with confidence ≥0.60.
• A.6 A RECORDED RUN follows “a three-legged wooden stool” in Three.js. Its score rises from 0.410 to 0.844 and then 0.933 as revisions connect parts and improve geometry, taking 21 model calls, 4.0 active minutes, and $0.74. The selected output still has an unmet footprint requirement, so the highest-scoring round does not satisfy every mandatory condition.
• A.7 ARTICULATED OBJECTS: GEOMETRY, KINEMATICS, AND JOINT SWEEPS couples Blender link meshes with a URDF kinematic tree. Build validation rejects rest-pose alignment discrepancies exceeding 1 mm or penetration exceeding 5 mm; dynamic checks test joint limits and eight seeded multi-joint configurations, flagging penetration exceeding 2 mm in moved poses. The laptop example detects a 2.5 mm hinge–base collision and caps an otherwise favorable score at 0.60.
• A.8 SCENES AND GRAPHICS describes staged asset and zone construction, escalating from fast drafts to interactive repair when needed. Scene staging is bounded at 55% of the wall-clock budget, with checks for placement, exposure, motion, and a 15,000 draw-call budget, limiting rendering commands. Graphics are rendered at t ∈ {0, 1, 2.5, 4, 6} s, clipped to the loop duration, at 1280 × 720 resolution, with frame checks for invalid pixels, black or overexposed output, static sequences, and flicker.
• A.9 SKILLS documents seventeen authoring bundles, reusable packages of instructions and code references, with each reference bounded at 350 lines and approximately 2,500 tokens. A 28-rule router selects up to five bundles per session using the active task, runtime, planned structure, and preceding diagnostic findings.
• B 3DCODEVERSE-2M: DATASET DETAILS distinguishes the packaged program catalogue, benchmark training pairs, and expanded text exports. These are different counting units, and individual fine-tuning experiments use specified mixtures drawn from the available records.
• B.1 DATASET HIERARCHY AND COUNTING UNITS reports 269,597 unique executable programs, 768,596 deduplicated benchmark training pairs containing 891M tokens, and 2,028,286 unified export rows across 342 shards. The export comprises 891,788 native-format rows, 848,819 standardized text-export rows, and 287,679 shader rows that separate multi-pass programs into individual rendering-pass targets. Benchmark training splits exclude 3DCodeBench factories, seeded instances, and per-dialect evaluation suites.
• B.2 SOURCES AND CONVERSION PIPELINES reports 128,201 deduplicated executable DeepCAD conversions with mean voxel IoU 0.985 at 48³ resolution, and 6,878 of 6,882 Articraft CAD conversions passing geometric verification. Other sources include 119,961 Shadertoy shaders and 15,704 execution-verified harness bootstrap programs, generated examples used for subsequent training. Table 5 states dataset-level CC-BY-NC-SA-4.0 terms alongside source-specific terms, with the stricter applicable conditions controlling reuse.
• C EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS supplies training configurations, representation-scale loss curves, evaluation protocols, model-mixture comparisons, and harness component studies. It distinguishes decoding settings, training mixtures, refinement horizons, and evaluation subsets needed to interpret the main results.
• C.1 TRAINING DETAILS uses 4 × NVIDIA H100 (80 GB) GPUs in bfloat16 numerical precision, gradient checkpointing to reduce activation memory, and sequence packing to combine examples. The standard SFT recipe uses rank-64 LoRA with α=128 and dropout 0.05; DPO uses rank-16 LoRA with α=32, preference-loss parameter β=0.1, learning rate 5 × 10⁻⁶, and two epochs. Context limits are 8,192 tokens for 9B and 4,096 for 27B, with oversized examples excluded rather than truncated.
• C.2 TRAINING CURVES OF THE 27B DATA-SCALE RUNS compares three one-epoch rank-16 LoRA runs using different export representations. Arms A, B, and C use 1,992,816, 1,160,630, and 292,690 rows, respectively, and finish at smoothed training losses of 0.274, 0.314, and 0.292. These losses measure prediction error on different training mixtures; the curves alone do not establish differences in held-out generation quality.
• C.3 EVALUATION PROTOCOL specifies deterministic code extraction, a 300 s execution timeout, unit-normalized meshes, 10,000 sampled surface points, and best alignment over four 90° yaw rotations for geometric evaluation. Harness comparisons use a single user request without subsequent human edits, while allowing internal tools and repair. The fixed evaluator rebuilds outputs independently, and pairwise judgments use anonymized views with randomized human presentation order and agreement across both display orders for VLM preferences.
• C.4 ADDITIONAL OPEN-MODEL RESULTS shows that training mixture and preference signal affect different aspects of performance. Two execution-feedback DPO rounds raise greedy 3DCodeBench execution, using the highest-probability token at each step, for the 9B 5.2K checkpoint from the reported 75.0% baseline to 96.2%; pass@4, success in at least one of four sampled completions, rises from 91.0% to 99.1% at decoding temperature T=0.7. Geometry-feedback DPO improves shape fidelity conditional on successful execution, while termination-feedback DPO reduces repetitive output; improvements are not uniform across checkpoints or metrics.
• C.5 ADDITIONAL HARNESS RESULTS reports paired gains of +15.5 to +52.3 points in separate studies with 8–11 prompts per configuration. An explicit joint-coupling planning field raises valid URDF mimic declarations, which specify one joint’s motion as a function of another’s, from 3/10 to 10/10; component telemetry reports invalid plans falling from 4.7% to 0.7% and redundant retries from 26.6% to 1.3%. Auxiliary results elsewhere in Appendix C also show exceptions: Table 9 reports a −5.4-point mean graphics change for Claude Sonnet 5, with a 95% confidence interval of [−20.8, 10.1].
• D EXECUTABLE ARTICULATION AND COMPARISON WITH UWM evaluates two assets × two initial states × three control regimes using 16 commands at 20 Hz, with three UWM seeds per case. Direct simulation and VAE-reconstructed renders each preserve assessable contours in 12/12 cases, versus 7/36 for UWM rollouts. The static last-frame control also passes 12/12 without motion, so the result measures rigid-geometry preservation under this setup rather than overall dynamic-prediction quality.

Brief Thoughts

The combination of executable source, measured structural feedback, and reusable training records is the paper’s most useful contribution. Numerical corrections and measurement-based score constraints give agents a concrete way to repair defects that rendered appearance can miss, and the training and human-preference results support the integrated workflow.

The evidence supports workflow-level quality gains more clearly than compute efficiency or uniform improvement. Generation costs increase substantially, some auxiliary studies are small, verification varies by source, and main-table values disagree with the appendix. Equal-budget comparisons, reconciled result tables, and broader repeated-run evaluation would help determine the practical trade-offs.