Generating Physically Stable and Buildable Brick Structures from Text | Summary

This article explains the key points of Generating Physically Stable and Buildable Brick Structures from Text.

• 2025-05-08 (arXiv)
• Pun, Ava, Deng, Kangle, Liu, Ruixuan, Ramanan, Deva, Liu, Changliu, Zhu, Jun-Yan.
• Carnegie Mellon University
• Paper
• Github
• Project Page

Read this article in Korean

Summary

• Generating Physically Stable and Buildable Brick Structures from Text addresses how to turn a written object description into an interlocking toy-brick design that can be assembled without collapsing on a baseplate. On 250 validation prompts, BRICKGPT produces 100% valid designs, with legal pieces, in-bounds placement, and no collisions, and 98.8% stable designs, compared with 100% validity and 88.4% stability for Hunyuan3D-2 followed by conversion of generated 3D surfaces into bricks and stability-based selection among up to 100 layouts.
• BRICKGPT fine-tunes a large language model, a model trained to predict text sequences, to generate brick dimensions and positions autoregressively, meaning one brick at a time conditioned on earlier bricks. StableText2Brick supplies over 47,000 stable structures from over 28,000 objects across 21 categories; inference combines rejection sampling, which resamples invalid placements, with physics-aware rollback, which removes an unstable suffix and resumes generation.
• The ablations separate geometric validity from physical stability: rejecting invalid bricks raises validity from 37.2% to 100%, while adding rollback raises stability from 24.0% to 98.8%. Manual and robotic assembly demonstrate physical realizability for selected designs, but the method remains restricted to a 20 × 20 × 20 grid and eight standard brick types.

1. Introduction

The paper distinguishes physical stability, meaning that a structure on a baseplate has no floating or collapsing bricks, from buildability, meaning compatibility with standard pieces and assembly by humans or robots. Existing text-to-3D generation, which creates three-dimensional objects from written descriptions, can produce convincing geometry without satisfying either requirement, while earlier brick-design methods commonly require an existing 3D shape or target a single category.

• BRICKGPT represents design as next-brick prediction, with a textual record specifying each brick’s dimensions and placement.
• Figure 1 connects generated designs to manual and robotic assembly and illustrates separate appearance generation for an unchanged brick geometry.

The vessel example connects a text prompt to incremental brick placement and physical assembly. The gallery also separates structural generation from later appearance changes, showing the range of demonstrated outputs without supplying a quantitative success rate.

2. Related Work

The related work covers Text-to-3D Generation, Autoregressive 3D Modeling, Brick Assembly and Design Generation, and Physics-Aware Generation. These areas address shape synthesis, sequential geometric prediction, component arrangement, and physical constraints; BRICKGPT combines sequential prediction with checks for interlocking brick structures.

• Mesh generation methods such as LLaMA-Mesh produce surfaces represented by vertices and faces. They motivate a plain-text geometric representation but do not directly enforce brick compatibility or stability.
• Shape-to-brick optimization and assembly planning usually begin with a target shape, whereas BRICKGPT directly generates the brick structure from text.
• The paper distinguishes simple geometric checks, such as collision avoidance, from structural stability analysis that requires reasoning about forces.

3. Dataset

StableText2Brick contains 47,000+ stable brick structures representing 28,000+ unique objects from ShapeNetCore, a dataset of 3D object shapes, across 21 categories. A structure is a sequence of bricks, each described by its two horizontal dimensions and three coordinates, with the position anchored at the stud, the raised connection knob, closest to the origin.

• Dataset construction voxelizes each mesh, converting its occupied volume into cells of a 20 × 20 × 20 discrete grid, then uses a randomized delete-and-rebuild algorithm to produce alternative brick layouts.
• A nonlinear force analysis assigns each brick a stability score between 0 and 1; only structures whose every brick has a score greater than 0 enter the dataset. Analysis averages approximately 0.35 seconds for the illustrated structures containing fewer than 200 bricks.
• GPT-4o receives renderings from 24 viewpoints and generates five geometry-focused captions at different detail levels, omitting color and texture.

The pipeline shows that stable training examples are constructed through shape conversion, randomized layouts, and explicit stability filtering. Captions describe multiview renderings of the resulting brick structures, aligning language supervision with the discretized designs.

4. Method

The method combines instruction fine-tuning, training on requests paired with desired responses, with constrained generation at inference. Figure 3 shows the brick sequence representation, training setup, inexpensive placement checks, and final stability verification followed by rollback when needed.

The diagram distinguishes local checks during brick prediction from physical verification after a complete design. A collision or unavailable piece type triggers resampling, whereas physical instability triggers removal of a suffix and renewed generation.

4.1. Model Fine-tuning

BRICKGPT fine-tunes LLaMA-3.2-1B-Instruct to output one brick per line in the format “{h}×{w} ({x},{y},{z})”. Every brick is a 1-unit-tall, axis-aligned cuboid; exchanging the horizontal dimensions encodes its orientation, and the training sequence follows a raster scan, a systematic spatial traversal, from bottom to top.

• The custom representation includes dimensions needed for placement validation while omitting arbitrary orientation and scale information present in LDraw, an existing brick-model format.
• Instruction examples pair an object caption with its complete brick sequence, allowing reuse of the language model’s tokenizer, which converts text into model input units.
• The pretrained model can produce partially structured outputs through in-context learning, conditioning on examples supplied in the prompt, but collisions, disconnected components, and poor prompt adherence motivate task-specific fine-tuning.

4.2. Integrating Physical Stability

Physical stability is evaluated through static equilibrium, the condition that every brick’s net force and torque balance. The force model includes gravity, vertical connection forces, horizontal forces at stud connections, and contact with adjacent bricks; Gurobi, an optimization solver, finds a force distribution subject to nonnegative force magnitudes, incompatible-force exclusions, and Newton’s third law of equal and opposite interaction forces.

• A brick receives score 0 when equilibrium cannot be achieved or required connection friction exceeds the measured capacity; otherwise, its score reflects the remaining friction margin. A complete structure is stable only when every score is greater than 0.
• Brick-by-brick rejection sampling checks formatting, inventory membership, world bounds, and collisions, then resamples a placement that fails these checks.
• Physics-aware rollback checks the completed design, removes the earliest unstable brick and every subsequent brick, and repeats removal until a stable prefix remains before resuming generation. Checking full stability after every added brick would be slower and could exclude structures that become stable only after later additions; the implemented rollback limit also permits termination with an unstable output.

The illustrated forces show why collision-free placement is insufficient for stability. The solver must balance gravity and connection forces across multiple bricks while respecting the capacity of their connections.

4.3. Brick Texturing and Coloring

Appearance generation operates after structural generation through either UV texturing, which maps a 2D texture onto a mesh surface, or a uniform color assigned to each brick. FlashTex, a text-conditioned mesh-texturing method, generates the appearance from an additional prompt.

• For detailed textures, fully occluded bricks are removed, visible bricks are merged using ImportLDraw, a tool for importing LDraw models, and cube projection creates surface texture coordinates. The paper proposes UV printing or stickers for transferring these textures to physical pieces.
• For standard colored bricks, texture colors are averaged over visible voxel faces and then over each brick’s constituent voxels; the nearest color in a standard library is selected.
• Figure 7 demonstrates several appearances for the same underlying geometry.

5. Experiments

The experiments evaluate whether generated structures use legal, noncolliding placements, remain physically stable, and match the requested object. Comparisons include pretrained language-model prompting, text-to-3D methods followed by brick conversion, and ablations that remove BRICKGPT’s inference constraints.

5.1. Implementation Details

Fine-tuning uses 240k distinct prompts and 47k+ distinct structures, with 10% of the data withheld for evaluation and training examples limited to 4096 tokens. Evaluation generates one structure for each of 250 randomly selected validation prompts.

• The stability solver uses a connection friction capacity of 0.98 N, with optimization weights α = 10⁻³ and β = 10⁻⁶.
• Generation permits up to 100 physics-aware rollbacks before accepting the structure. The median run requires 2 rollbacks and 40.8 seconds per structure.
• The appendix reports that 1.2% of generated designs exceed the rollback limit and fail to produce a stable final structure.

5.2. Brick Structure Generation Results

Table 1 reports 100% validity, 98.8% stability, mean brick stability of 0.996, and minimum-brick stability of 0.915 for BRICKGPT. Its CLIP score, a rendered-image/text alignment measure, is 0.324, and its DINOv2 score, a visual-feature similarity measure against the corresponding reference structure, is 0.880.

• The strongest stability baseline is Hunyuan3D-2 plus stability analysis at 88.4%, with CLIP 0.324 and DINOv2 0.868. Baselines first generate meshes and convert them into bricks; stability-based selection tests up to 10 layouts per mesh for LLaMA-Mesh and LGM and up to 100 for XCube and Hunyuan3D-2, accepting the first stable layout found.
• Without rejection sampling or rollback, BRICKGPT reaches 37.2% validity and 12.8% stability; with rejection sampling but no rollback, it reaches 100% validity and 24.0% stability. Figures 5 and 6 illustrate collisions and structural failures.
• Section 5.2 describes validity and stability as proportions among generated designs, while the Table 1 caption says stability, CLIP, and DINO are computed over valid structures only. These descriptions leave the stability denominator inconsistent, so the reported percentages are retained without resolving that discrepancy. Table 1 applies a special validity criterion to LLaMA-Mesh: a well-formed OBJ surface-mesh file. The unconstrained ablation has higher CLIP and DINO scores, 0.329 and 0.888, so the full method’s principal improvement is physical feasibility while retaining comparable similarity.
• The pretrained language-model baseline achieves 0.0% validity with zero examples and 2.4% with five examples. These prompting baselines omit rejection sampling and rollback because the authors report excessive rejections and sharply increased generation time.

BRICKGPT reaches 98.8% stability versus 88.4% for the strongest stability-selected baseline. Removing rollback reduces stability to 24.0% despite retaining 100% validity, while the unconstrained ablation’s higher similarity scores show that physical feasibility and visual alignment are separate evaluation dimensions.

Matched prompts expose different failure modes: pretrained prompting often produces collisions, while mesh-conversion pipelines can produce recognizable but unstable objects. The selected BRICKGPT examples are stable, complementing the aggregate results without representing every evaluation output.

The chair and vehicle examples illustrate the different roles of the two inference mechanisms. Rejection sampling removes collisions, and adding rollback produces the stable structures shown in the final column.

5.3. Extensions and Applications

The paper demonstrates manual assembly, dual-arm robotic assembly, and text-driven appearance changes. These examples show that selected digital designs can be realized with physical bricks and that appearance can vary while preserving the generated shape.

• Robotic execution uses a separate manipulation policy and asynchronous planner, which coordinates the two arms’ tasks and motions.
• A separate assembly planner reorders bricks so that intermediate structures remain stable, withstand robot operation, and permit executable placements; final-design stability alone does not establish those properties.
• The reported assembly evidence consists of demonstrations rather than an aggregate assembly success rate over the 250-prompt evaluation.

The bookshelf and sofa retain their geometry across detailed appearance prompts, while the guitar receives different uniform brick colors. Detailed surface textures would require an additional physical treatment such as printing or stickers, whereas uniform coloring maps to standard colored pieces.

6. Conclusion

The paper concludes that next-brick generation, stable training data, and explicit inference-time verification together improve the physical feasibility of text-generated brick structures. Supplementary evidence includes few-shot generation failures, nearest-training-design comparisons, and robotic and manual assembly examples. The dataset, code, and models are provided at https://avalovelace1.github.io/BrickGPT/.

The few-shot example produces a recognizable partial arrangement but includes collisions and disconnected pieces. It illustrates that conditioning on example structures does not by itself produce a physically valid design.

Generated designs differ visibly from their closest training examples under voxel-space Chamfer distance. These selected comparisons provide evidence against exact shape copying in the displayed cases, without quantifying novelty over the full evaluation set.

The vessel and guitar sequences demonstrate completed physical builds using two robot arms. Their execution relies on separate assembly planning and force-controlled manipulation in addition to the generated final geometry.

Paired renderings and physical builds show that selected designs across several categories can be assembled with real bricks. The gallery supplies broader qualitative assembly evidence, but does not report an assembly success rate or construction-time distribution.

Appendix

• A. BRICKGPT Implementation Details: Dataset construction produces 62,000+ layouts from 31,218 ShapeNetCore objects, generating two layouts per object; approximately 92%, or 28,822 objects, have at least one stable layout. The eight supported brick types are 1 × 1, 1 × 2, 1 × 4, 1 × 6, 1 × 8, 2 × 2, 2 × 4, and 2 × 6. Training uses low-rank adaptation (LoRA), which trains small low-rank parameter updates, on query and value matrices in the model’s attention layers, with rank 32, alpha 16, dropout 0.05, and 3.4M trainable parameters. Three epochs, or passes through the training data, on eight NVIDIA RTX A6000 GPUs take 12 hours, using AdamW, an optimizer with decoupled weight decay, with learning rate 0.002, a cosine learning-rate schedule, 100 warmup steps, and global batch size 64. Inference starts at temperature 0.6, which controls sampling randomness, and increases it by 0.01 when an already rejected brick recurs; token restrictions enforce the output grammar. Figure 8 illustrates few-shot collisions, while Figure 9 compares generated shapes with nearest training shapes using Chamfer distance, a nearest-neighbor geometric distance calculated here in voxel space.
• B. Robotic Assembly: Two Yaskawa GP4 robots, each equipped with an ATI force-torque sensor, assemble bricks on a calibrated baseplate. Action masking, which excludes unsuitable assembly actions, and assembly-by-disassembly search, which derives a construction order from a disassembly search, produce an executable sequence. The sequence must preserve intermediate stability, withstand robot operation, and respect the system’s dexterity; an asynchronous planner and closed-loop force control, which uses measured forces during manipulation, then coordinate execution.
• C. Manual Assembly: Figure 11 pairs generated renderings with physical builds across furniture, guitars, vessels, vehicles, and containers. The generated sequence supplies intermediate steps that can serve as a manual assembly guide.
• D. Limitations: The model is restricted to a 20 × 20 × 20 grid, 21 object categories, and a fixed library of common cuboid bricks. The authors propose larger and more diverse training datasets, higher grid resolutions, and additional pieces such as slopes and tiles; the paper does not experimentally establish these extensions or their effect on unfamiliar prompts.

Brief Thoughts

The clearest contribution is the measured separation between legal placement and structural stability: 100% validity without rollback still yields only 24.0% stable designs. The improvement to 98.8% supports using a physical verifier alongside learned generation in this setting, while the remaining failures show that the bounded inference procedure does not guarantee stability for every output. Interpretation should remain tied to the stated benchmark: 250 validation prompts, a fixed grid, standard cuboid pieces, and a static baseplate force model. The nearest-neighbor examples show geometric differences from training shapes, but no aggregate novelty statistic is reported. The paper also does not specify whether the 10% holdout separates underlying source objects, which limits assessment of generalization to unseen objects. Baseline inference differs materially: mesh methods receive repeated layout selection, while pretrained prompting lacks rejection sampling and rollback. Table 1 measures these complete configurations and does not isolate model architecture.