TrueMuse: A Benchmark for Data Attribution in Text-to-Music Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "TrueMuse: A Benchmark for Data Attribution in Text-to-Music Models".

Tom: The gist The introduction proposes TRUEMUSE, a controlled dataset and benchmark for text-to-music data attribution, designed to systematically evaluate existing black-box attribution methods across four musical attributes.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Now let's talk about what the paper actually claims they did. They are addressing a big gap where existing attribution methods struggle because there isn't reliable ground truth for text-to-music generation yet.

Jane: So the core idea is that they propose TRUEMUSE as this controlled dataset and benchmark to systematically evaluate black-box attribution methods against these known fine-tuning source sets.

Lu: The paper explains that white-box methods try to approximate sample contributions through parameter estimation, which doesn't tell you if a sample really mattered.

Meng: But black-box methods operate on models trained on massive datasets where many samples influence the parameters at once, making it hard to isolate one specific contribution.

Tom: That entanglement makes it really difficult to figure out the exact influence of an individual sample on a given music generation. So they need this controlled benchmark to test their effectiveness.

Jane: They are constructing TRUEMUSE by fine-tuning three diffusion-based text-to-music models, AudioLDM2, Mustango, and Stable Audio Open, on these specific attribution samples they have curated.

Lu: The construction process adapts concept-specific fine-tuning from text-to-image diffusion models to text-to-music generation by binding the target musical attributes to selected audio samples.

Tom: It’s like they are taking that idea from image generation and applying it here, making sure those source clips are explicitly known targets for the attribution evaluation.

Jane: This setup allows them to generate samples using both contextual and style prompts, which is important because we need to test how the attribution works under different kinds of instructions.

Meng: So they are generating these ninety-five thousand five hundred forty-six samples across those two prompt types, and then using that output to see how well an attribution method can retrieve the right source clip <ref:2610.00835#pg1>.

Tom: And they set up this retrieval task where a generated sample is the query and it has to be matched against a gallery of clips from the designated source set.

The paper's summary: Jane: Moving into what they suggest are the improvements in their approach, they point out that existing methods can be rigorously evaluated across four dimensions.

Tom: Those dimensions are fine-tuning improvement, prompt-type difficulty, multitask training, and fine-tuning data size. They want to see which methods are more reliable and generalizable based on these tests.

Lu: They evaluate similarity-based black-box attribution using four different music encoders: MERTv0, MERTv0 public, DAC, and Music2Vec.

Meng: So they aren't just testing one tool; they are checking how these different types of encoders perform when trying to attribute music back to its sources.

Tom: And the results show that attribution remains challenging because existing methods vary quite a bit depending on which setting you test them in. It’s not straightforward.

Jane: The paper also introduces a new method where they use a mapper, which is essentially a linear projection head applied to the output embeddings of the frozen music encoder Fbase.

Lu: This mapper is trained with a contrastive loss using generated samples and their ground-truth clips to create embeddings that are better suited for attribution retrieval.

Tom: This mapper specifically improves performance for MERT-based encoders, showing gains like MERTv0 improving from zero point three five seven to zero point four zero zero in some cases.

Meng: But the gains aren't universal; they're largest on Mustango, where it gets a jump of about eight point four percent for both MERTv0 and MERTv0 public, while the improvement on Stable Audio Open is minimal, only about one point five percent.

The paper's improvements: Tom: So wrapping up this part of the paper, they show that style prompts actually outperform contextual prompts on average across all those different encoders they tested.

Jane: For example, for Melody-Centric attribution, style prompts increase retrieval performance from a score of zero point one seven eight under contextual to zero point three six five under style prompts.

Lu: This suggests that style prompts provide a cleaner and more attribute-focused training signal for the mapper they are training, which is quite useful for guiding the model's learning process.

Meng: It seems like they found that if you want a better attribution, you should be thinking about how the prompt is designed to focus on a specific musical feature rather than just describing what’s happening in context.

Tom: And they also looked at increasing the number of fine-tuning clips from three to six, which consistently reduced retrieval scores across the Instrument-Centric, Musician-Centric, and Genre-Centric settings.

Jane: That's because adding more clips expands the candidate source set, and this can make individual clips harder to retrieve when you increase that count.

Lu: So they conclude that source retrieval remains challenging and varies substantially across musical attributes and generators like Stable Audio Open compared to others like Mustango or AudioLDM2.

Meng: As an initial benchmark, TrueMuse focuses on a representative set of musical attributes, generators, and prompt conditions; future work could expand the coverage of models and concepts further.

Tom: This whole paper on TrueMuse gives us a controlled environment to study data attribution in text-to-music systems without reliable ground truth right now. It’s a solid foundation for where researchers can go next.

Jane: Indeed, it sets up a framework for more rigorous study of how we assign credit in these complex generative systems and supports developing methods that are more robust and interpretable.

Conclusion: Tom: So TrueMuse is this new benchmark for data attribution in text-to-music models, and what they did is build this controlled dataset where they know exactly which training samples are included in the fine-tuning.

Jane: Exactly, so it lets researchers systematically test how well existing black-box methods work when they have these known source sets to check against.

Lu: The construction is pretty interesting because they're taking that concept from text-to-image diffusion and applying it here to music generation.

Meng: It’s a controlled environment, which is what we need when trying to figure out if an attribution method actually points to the right training data contribution.

Lalam: If this works well, it could really help us understand how culture itself gets encoded in these generative models.

Tom: They are testing things across four major musical attributes: melodic structure, timbral characteristics, artist-level signatures, and genre patterns.

Jane: And they're using these generated samples to evaluate retrieval tasks where the sample is the query and the source clips are in a gallery based on which attribute it was fine-tuned on.

Lu: The results show that attribution remains pretty tough with current methods because there's so much variation depending on what you test them on.

Meng: They did look at how different encoders like MERTv0 and DAC perform, and they found that MERTv0 public was the strongest performer in some cases for R@ten scores.

Tom: And they also showed that the prompt type matters; style prompts are better than contextual ones on average across all those encoders.

Jane: That makes sense, because style prompts seem to give a cleaner signal for the mapper to learn from.

Lu: They also found that increasing the number of fine-tuning clips generally made retrieval harder in settings like Musician-Centric.

Meng: So it’s not just about getting more data; it’s about how that extra data affects the difficulty of isolating a single sample.

Lalam: It gives us a way to audit what's happening inside these massive models, which is super important for understanding their cultural output.

Tom: This whole TrueMuse benchmark is a huge step because it provides that controlled environment for evaluating attribution methods in text-to-music systems.

Jane: It helps move the conversation from just making cool music to actually understanding what's driving the music generation process.

Lu: It's a solid foundation, and they’ve laid out exactly where future research needs to go if we want better credit assignment.

Meng: I think seeing how these methods perform under different prompt types is really practical for deployment later on when we try to use AI for specific creative tasks.

Lalam: It opens up a lot of possibilities for how we can develop systems that are more transparent about their training data influences.