COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing

summary

Video file (mp4)

The gist

The gist: This paper proposes a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently to achieve precise and disentangled editing of

In short

The paper introduces a new generator architecture for 3D Gaussian Splatting (3DGS) that synthesizes human head components—like hair, skin, glasses, and torso—independently. This allows users to edit one feature without affecting others. It uses minimal information, like color histograms and context tokens, to maintain consistent shape and lighting during editing.

Key concepts

Compositional GAN Architecture
This architecture builds the main generator by combining several smaller, specialized sub-generators, one for each part of the head (e.g., hair or skin). Each sub-generator focuses only on its assigned region, enabling independent control over that specific attribute during synthesis.
Additive Components
For optional parts like glasses, a boolean flag controls their presence. If present, the component is generated and added to the final scene; if absent, its opacity is reduced. This ensures the generator learns to exclusively create only what is requested by that specific sub-generator.
Shape Context
A shared low-dimensional latent vector provides a global signal to all sub-generators during training. This context ensures that when components are mixed or edited, the overall head shape and scale remain consistent, as this vector is only used to modify spatial coordinates.

Terminology used across episodes

This episode discusses

The paper

COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing · Read on arXiv

Florian Barthel, Shalini De Mello, Koki Nagano, Wieland Morgenstern, Anna Hilsmann, Peter Eisert

Fraunhofer Heinrich-Hertz Institute

Recent 3D Gaussian Splatting (3DGS) GANs for human heads synthesize and render photorealistic 3D models in real-time and offer a vast variety in identity and appearance. However, controlling specific semantic attributes such as hair color or glasses remains challenging, as edits in the entangled latent space often induce unintended changes in identity or appearance. Although there are several methods that aim to disentangle the latent space post training by estimating directions that only modify certain features, these methods cannot guarantee complete disentanglement and often require pre-trained classifiers. In our approach, we propose a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently. This allows for changing the latent vector for one region while keeping the remaining parts fixed. Further, we achieve this separation using only sparse information such as the hair or skin color, eliminating the requirement of segmentation masks or geometric priors, often seen in prior work. To ensure matching shape and lighting conditions during editing, we allow minimal shared information via context tokens between the independent generators. These tokens even allow us to control the shape and light, without any prior annotation. Compared to existing works on GAN-based generation and editing, our method shows better disentanglement, more precise editing control, and competitive visual quality.

DOI: 10.1007/978-3-032-37138-6_22

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing".

Jane: The gist: This paper proposes a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we’re talking about this paper called "COSY: Compositional three deeGS Synthesis for Disentangled Human Head Editing <ref:2605.24114#pg1,COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing>." It's tackling a big problem in three dee modeling where you can make photorealistic human heads in real-time, but it’s hard to change just one thing without messing up the whole face <ref:2605.24114#pg1>.

Jane: Exactly. The paper says that when you edit something like hair color or adding glasses, those edits often cause unintended changes to the person's identity or appearance because the latent space is too tangled.

Lu: The core idea of COSY is proposing a new generator architecture that builds the head from completely independent components—hair, skin, glasses, and torso—so you can change one part without affecting the others.

Meng: So it’s not just another way to train a GAN; it’s about separating the three dee model into these distinct parts so we can manipulate them individually during editing <ref:2605.24114#pg1>.

Lalam: It claims that this approach allows for precise and real-time editing at high resolution, which is a big deal compared to what was done in prior work on real-time GAN editing <ref:2605.24114#pg3>.

Tom: And they claim this decomposition is achieved without needing explicit segmentation masks or geometric priors, which they say is different from some earlier methods that required those things.

Jane: That’s important because getting precise masks can be really difficult in practice, and COSY seems to skip that step by using sparse information like just the hair or skin color.

Lu: They use this sparse information to separate the components, and then they introduce shared context tokens between these independent generators to keep things consistent when you're editing.

Meng: So those context tokens are what allow them to control shape and lighting without needing any prior annotation for that part of the process.

Lalam: It sounds like they also have training strategies built in to make sure that when you mix different components together during inference, the quality stays high.

Tom: Which brings us into the conclusion of this paper. The authors discuss what this means for how we think about editing these three dee models in general <ref:2605.24114#pg1>.

Jane: They are focusing on achieving a structure where you can edit specific attributes with much finer control over the result than before, which they call disentangled editing <ref:2605.24114#pg1>.

Lu: The title itself points to the key contribution, suggesting that this compositional approach provides an independent latent space for hair, face, glasses, and torso to allow for precise and disentangled editing of three dee human head avatars <ref:2605.24114#pg1>.

Meng: For practical use, it means a creator or a developer can now change just the skin tone or the hairstyle without having to worry about accidentally altering their facial structure during that edit.

Tom: And they also mention how they ensure consistency through shared shape and lighting context latent vectors that stay consistent across all sub-generators during inference <ref:2605.24114#pg3>.

Jane: It suggests that even when you're making complex edits, the overall head shape and the way it's lit stays stable because of these shared signals.

Lu: The paper’s contribution is really this decomposition into semantically localized sub-generators without needing explicit annotation <ref:2605.24114#pg3>.

Meng: If this works well, it opens up possibilities for synthetic data generation where you can create highly specific three dee assets with fine control over their appearance <ref:2605.24114#pg1>.

Lalam: And from an AI perspective, this architectural separation could help build more modular and controllable generative systems that are easier to manage and fine-tune <ref:2605.24114#pg3>.

Tom: So, what does this mean for the future of three dee avatar creation <ref:2605.24114#pg1>? It seems like we move closer to a point where editing these models becomes truly intuitive and precise.

Conclusion: Tom: So we’re wrapping up this look at COSY, which is "Compositional three deeGS Synthesis for Disentangled Human Head Editing." Basically, this paper lays out a new way to build and edit three dee human heads where you can change one part without messing up the rest.

Jane: Yeah, it moves away from just having one big tangled latent space and instead breaks the head down into separate generators for hair, skin, glasses, and the torso.

Lu: It’s interesting how they tackle that disentanglement by using these sub-generators that only look at specific parts of the data when they are trained.

Meng: So for someone building an application, what does this decomposition actually mean in terms of control? How much control are we talking about?

Lalam: It means you get granular control. You can adjust the color of just the hair, for example, without accidentally shifting the person's facial structure because it’s handled by a separate component.

Tom: Exactly. The authors claim this method gives more precise editing and better disentanglement than what’s been done before on GAN-based models.

Jane: It also tackles a big hurdle with segmentation masks. They manage to do this separation using only sparse information, like just the color of the hair or skin, instead of needing detailed geometric masks.

Lu: And they use these context tokens between the generators to keep things consistent when you're doing an edit during inference, which helps maintain shape and lighting.

Tom: So it seems they’ve built a system that separates identity from appearance features more effectively than previous approaches.

Jane: It really suggests a future where three dee avatar editing becomes much more intuitive for users because the components actually understand what they are supposed to be manipulating.

Lu: This opens up possibilities for creating highly controllable synthetic assets, which is huge for things like virtual reality content or training data.

Meng: I’m curious about the practical side—how does this architecture run in real-time? Can we expect it to maintain the speed of Gaussian Splatting rendering while doing this decomposition?

Tom: That's what they test. They show competitive visual quality while achieving that separation, and it seems they have a solid foundation for that.

Jane: It’s a lot of technical heavy lifting to get these sub-generators to play nice, but the potential for better editing control is definitely something worth watching closely.

More episodes

← Home