BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization

arXiv:2605.26182 · cs.AI, cs.GR · Submitted 2026-08-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization".

Jane: The paper was written by Zhengyang Ni, Feng Yan, Yu Guo and Fei Wang from Xi'an Jiaotong University and State Key Laboratory of Human-Machine Hybrid Augmented Intelligence and Institute of Artificial Intelligence and Robotics.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Alright, listeners, welcome back to the show. I’m Tom, and with me as always is the brilliant Jane. Jane, we’ve got a paper today that’s got me genuinely excited, and the title alone is a mouthful: "BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization."

Jane: Tom, I love this one. And for anyone just tuning in, this is all about getting a computer to turn a three dee shape into a structure made of LEGO-style bricks. Not just a picture that looks like the shape, but an actual, physically buildable model where every brick snaps together properly.

Tom: Exactly. And the name "BrickAnything" is a promise, right? They want it to work on any three dee shape you give it, whether that shape comes from a three dee scan, a design file, or even a text prompt.

Jane: Right. And the clever part is in the subtitle: "Structure-Aware Tokenization." That’s the secret sauce. Instead of just listing bricks in a boring order, they organize them like a family tree, where each brick is attached to a parent brick. It’s a much more natural way to think about construction.

Tom: So it’s not just about the bricks themselves, but how they connect. That’s the "structure-aware" part. And I have to say, seeing a computer generate a stable brick model that actually stands up is way more impressive than just generating a mesh.

Jane: Totally. A mesh is just a surface, but a brick structure has to obey gravity and the rules of the brick system. It’s a much harder problem, and this paper tackles it head-on. I can’t wait to get into the details of how they actually do it.

Tom: And I want to know if it really works on the hard cases, the shapes that trip up other methods. Let’s get into the summary next.

Summary: Jane: So, Tom, let’s break down what this paper actually does. The core problem is that old methods for brick generation either use a lot of trial-and-error searching, which is slow and breaks on hard shapes, or they use AI to generate bricks but don't give the AI enough information about the three dee shape it's supposed to be building.

Tom: Right, so some of the earlier AI methods just look at a text prompt like "a red car" and try to build one, without really knowing the car's exact dimensions. This paper fixes that by using a point cloud as the guide.

Jane: Exactly. A point cloud is just a bunch of dots in three dee space that trace the surface of the shape. It’s like a map. The AI looks at this map and then starts placing bricks, one by one, to fill in the shape.

Tom: And that’s where the "tree tokenization" comes in. Instead of writing out a list of bricks with their absolute coordinates, they write out a sequence that says, "This brick is attached to that brick, on this side, and it’s this size." It’s like giving someone directions to your house by saying "turn left at the big oak tree" instead of giving them GPS coordinates.

Jane: That’s a great analogy. It makes the generation process a lot more logical. The AI is basically building a family tree of bricks, and it’s much easier for it to learn the rules of stability that way. They also added a post-training step to make the structures more stable and a clever rollback system for when things go wrong.

Tom: And the results? They show that their method, BrickAnything, is much better at creating stable structures than the older search-based method, especially on those tricky shapes where the old method just gives up completely.

Jane: Right. The old method, Legolization, got a zero percent success rate on their "challenging" test set. BrickAnything, on the other hand, managed to create stable structures over eighty-three percent of the time. That’s a huge jump.

Tom: Wow, zero percent. That really shows the old method was hitting a wall. So the new approach isn't just a little better, it's solving a problem the old one couldn't even touch. So, what’s the real-world impact of this? Let’s talk about the improvements and where this could go.

Improvements: Tom: So, Jane, we’ve established that BrickAnything is a big step forward. But what are the real-world improvements here? What does this actually unlock?

Jane: Well, Tom, think about the practical side. The paper mentions they can extend this to image-to-brick and text-to-brick generation. You could type "a small, sturdy stool" and get a set of instructions for a brick model that actually holds its own weight.

Tom: That’s a great point. It’s not just for fun, either. This could be huge for rapid prototyping. Designers could use this to quickly create physical models of their three dee designs to see how they feel in the real world, without having to three dee print everything.

Jane: And I think the improvements in the AI’s "thinking" are just as important. By using that tree structure, they made the generation process much more efficient. The paper shows they need far fewer "rollbacks" – that’s when the AI has to undo a mistake and try again. It’s like the AI is making fewer wrong turns.

Tom: Fewer wrong turns means it’s faster and more reliable. And that’s what you need if you want to use this in a real product. You can’t have an AI that takes ten tries to build a simple chair.

Jane: Exactly. And the fact that they used a technique called DPO, or Direct Preference Optimization, is interesting. It’s a way to fine-tune the AI by showing it examples of "good" and "bad" brick structures, teaching it to prefer the stable ones. It’s a clever way to bake in the rules of physics.

Tom: So it’s learning from feedback, not just from a static dataset. That’s a powerful idea. I’m curious about the bigger picture here, though. Where does this kind of technology lead us in the long run? Let’s bring in the rest of the team for the final take.

Conclusion: Jane: We’ve covered a lot of ground on "BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization," so let’s bring in Lu, Meng, and Lalam to get their final thoughts.

Tom: Lu, you’re the visionary. What’s the big-picture impact?

Lu: This is a step towards bridging the digital and physical worlds more seamlessly. It’s not just about LEGOs. This concept of generating physically valid, discrete structures from geometry could apply to architecture, to modular robotics, even to designing custom furniture. The core idea of "structure-aware" generation is a powerful one.

Meng: I’m more focused on the engineering side, and what impresses me is the efficiency. The rollback numbers are dramatically lower than the baseline. That means less wasted computation and a more predictable system. That’s what makes this practical to deploy in a real application.

Tom: And Lalam, you’re the one who thinks about the cultural impact. What do you see?

Lalam: I see a democratization of creation. This technology could allow anyone, not just expert designers, to turn an idea into a physical, buildable object. It could change how we teach engineering and design, making it more hands-on and accessible. It’s about empowering people to build.

Jane: I love that. It takes a complex problem and makes it accessible. And that’s what great research does.

Tom: Couldn’t agree more. So, to wrap it up: "BrickAnything" shows us a smarter way to generate buildable structures by understanding the relationships between the parts. It’s a big win for geometric fidelity and physical stability, and it opens the door to a lot of exciting applications.

Jane: We’ll be watching to see where this goes. Thanks for joining us, everyone. That’s it for this paper, but we’ve got another fascinating one lined up for next time.

Tom: See you then, folks!

Zhengyang Ni, Feng Yan, Yu Guo, Fei Wang

Xi'an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence · Institute of Artificial Intelligence and Robotics

cs.AI, cs.GR

Submitted: 2026-08-16

Updated: 2026-08-18

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: The paper introduces BrickAnything, a unified geometry-conditioned framework for generating physically buildable brick structures from diverse 3D representations.

Key concepts

Structure-Aware Tokenization
This is the method used to describe how bricks connect. Instead of listing coordinates, the system writes out a sequence that specifies which brick is attached to a parent brick and on what side, making the generation process more logical like building a family tree.
Point Cloud
A point cloud is described as a collection of dots in three-dimensional space. It acts like a map that traces the surface of an object, guiding the AI as it places bricks to fill in the desired shape.
BrickAnything
This is the name of the method discussed, which aims to generate physical, buildable models using LEGO-style bricks. It can take various inputs, including 3D scans or text prompts, and ensure the resulting structure is stable.
Direct Preference Optimization (DPO)
DPO is a technique used to fine-tune the AI. It teaches the model to prefer stable brick structures by showing it examples of both 'good' and 'bad' designs, helping bake in the rules of physics.

Terminology

Summary

The paper introduces BrickAnything, a unified geometry-conditioned framework for generating physically buildable brick structures from diverse 3D representations. The authors state: "We present BrickAnything, a geometry-conditioned framework for buildable brick generation. By combining point-cloud conditioning with structure-aware tree tokenization, BrickAnything models local attachment relations in autoregressive generation."

The paper addresses the challenge that generating physically buildable brick structures from 3D shapes requires more than geometric reconstruction: the output must also satisfy discrete part constraints and structural stability. The authors note that existing methods fall into two categories: "heuristic optimization, which can break down when the target 3D shape does not admit a feasible structure under predefined constraints, or generate brick sequences without explicitly modeling the underlying 3D geometry and assembly relations. They pose the question: Can we design a unified framework for brick generation that explicitly leverages 3D geometry, adapts to diverse input representations, and generates buildable structures in a more flexible and effective manner?"

The authors adopt point clouds as a modality-agnostic intermediate representation, allowing diverse 3D inputs to be mapped into a common geometric interface. Point clouds provide explicit spatial information while being easily derived from a wide range of 3D modalities.

The core contribution is a structure-aware tree tokenization scheme that organizes bricks according to local attachment relations rather than spatial ordering. The authors construct a vertical attachment graph G = (B, E), where an edge indicates that two bricks are vertically adjacent and overlap in the xy-plane. They choose the root brick by the lexicographic order of (z, y, x) and perform breadth-first traversal over G. The root brick is encoded by absolute attributes (x0, y0, z0, h0, w0), while each non-root brick bi is encoded relative to its parent bp(i) as (fi, hi, wi, mi) where fi indexes the attachment position on the parent, (hi, wi) is the child brick size, and mi indexes the child-side anchor position. An EOP token marks the end of each parent's child group. This representation converts brick generation from global coordinate prediction into local attachment prediction.

The framework uses a pre-trained Michelangelo encoder E g to encode point clouds with normals into shape tokens, and OPT-350M as the autoregressive transformer. The model is trained with the standard next-token prediction objective.

The authors introduce a reward-guided DPO post-training stage with a buildability-aware reward that combines geometric fidelity and physical stability. Geometric fidelity combines voxel-level occupancy consistency (IoU) and surface-level geometric alignment (Chamfer Distance converted to a bounded reward). Stability is measured as the minimum per-brick score following StableLego. The overall reward is R = R geo + R stable ∈ [0, 3]. Preference pairs are constructed whose reward gap is at least 0.2 and whose higher reward is no less than 1. The DPO objective uses reward-weighted optimization with ∆R weighting, plus an auxiliary SFT loss to preserve data distribution.

During inference, the model samples a candidate tuple (f, h, w, m) which undergoes tuple-level validity checking including a feasible parent connector f, a brick size (h, w) from the predefined brick library, and a valid child-side anchor m, plus collision checking. For stability, the authors perform stability-guided rollback after a complete brick structure is generated. They identify the first unstable brick according to the generation order and trace its parent brick bp(k) in the generated tree structure and locate the token position where bp(k) itself is generated, then roll back to the state before generating bp(k) and regenerate the subsequent tokens.

The training data consists of approximately 230K high-quality meshes from ShapeNet, Objaverse, and Objaverse-XL, with around 168K stable mesh–brick pairs produced by Legolization. Evaluation uses two test subsets: a stable subset with 500 validation samples successfully converted by Legolization, and a challenging subset with 500 meshes sampled from cases where Legolization fails to produce stable structures. The workspace is a 20 × 20 × 20 voxel grid with eight commonly available standard LEGO® bricks: 1×1, 1×2, 1×4, 1×6, 1×8, 2×2, 2×4, and 2×6. Metrics include Voxel IoU, Chamfer Distance, structural validity percentage, stable generation rate, and average rollback count.

On the challenging subset, Legolization obtains 0.0% stable rates, indicating that heuristic search is brittle for geometrically difficult targets. The BrickGPT-style baseline with DPO achieves 76.0% stable rate and 0.553 IoU with 6.750 rollbacks. BrickAnything achieves 83.4% stable rate, 100% valid rate, 0.586 IoU, and a much lower rollback average of 0.422.

On the stable subset, Legolization obtains the best CD and IoU because this subset is selected from its successful cases. Among learned methods, BrickAnything improves IoU from 0.742 to 0.788, reduces CD from 0.1292 to 0.1283, and lowers the rollback average from 3.620 to 0.184, while maintaining 100% stable and valid rates.

Removing validity-constrained decoding, rollback, and DPO yields strong geometric scores but substantially weaker buildability, with the stable and valid rates dropping to 60.6% and 68.6% on the challenging subset. Adding validity-constrained decoding restores the valid rate to 100%, but the stable rate remains limited without rollback. Rollback improves stability by correcting unstable generations through structurally related resampling. DPO improves the generation distribution itself, improving IoU from 0.573 to 0.586 on the challenging subset and from 0.751 to 0.788 on the stable subset, while reducing rollbacks. The structure-aware tree tokenization consistently improves fidelity, stability, and correction efficiency compared to coordinate-ordered tokenization.

The authors note: BrickAnything is currently evaluated in a restricted 20 × 20 × 20 workspace with eight standard brick types, which constrains the resolution, part diversity, and geometric complexity of the generated structures. The representation is less effective for shapes whose geometry relies on slender components, subtle local variations, or smooth continuous surfaces. Future work will extend to larger workspaces, richer brick libraries, and more diverse connection types, incorporate human-designed brick models, and explore more adaptive reward models or incorporating real assembly feedback.

Improvements for AI systems

Based on the paper, here are the specific improvements I can make to AI systems and the resulting capabilities:

Improvement: Replace text/image-conditioned generation with a point-cloud-conditioned autoregressive transformer that uses explicit 3D geometry as the primary conditioning signal.

Capability: The system can generate physically buildable brick structures from any 3D input (mesh, point cloud, or even text/image via intermediate 3D reconstruction) while maintaining geometric fidelity to the target shape. Unlike text-only systems, this ensures the output is spatially aligned with the intended geometry rather than being semantically plausible but geometrically incorrect.

Net System Capability: An AI system incorporating these improvements can take any 3D shape (or image/text via intermediate reconstruction), generate a physically buildable brick structure that closely approximates the target geometry, maintain 100% structural validity, achieve high stability rates even on challenging geometries, and do so with minimal inference-time correction—making it suitable for real-time or resource-constrained applications where existing methods require excessive rollback or fail entirely.

Sources

Related papers