Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings

arXiv:2606.31602 · cs.CL, cs.CR · Submitted 2026-06-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings".

Jane: The paper was written by Jonas Schäfer, Cezary Pilaszewicz and Gerhard Wunder from Department of Mathematics and Computer Science and Freie Universität Berlin.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: We’re starting our discussion on the paper titled Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings, and this is a massive topic right now that everyone needs to understand.

Jane: To begin, this paper by Jonas Schäfer, Cezary Pilaszewicz, and Gerhard Wunder presents a fundamental shift in how we think about watermarking AI-generated text.

Lu: It’s not just hashing tokens like older methods; the authors are using sophisticated vector algebra to create a signal that is inherently tied to meaning rather than just structural patterns.

Meng: From an engineering standpoint, this "dual semantic embedding" approach means they are calculating biases at the inference-time logit level, which is the exact point where the AI decides what token to say next.

Lalam: That's a huge step because it implies that our model isn't just outputting random sequences; it’s making choices based on meaningful relationships that are preserved and detectable through semantic alignment.

Tom: So, we have this mechanism that ties the signal to meaning, but how do these two embeddings—the context one and the token one—actually work together?

Jane: Think of it as a sophisticated alignment check where they are asking how well the potential next word matches the overall context of what has already been written.

Lu: The mathematical elegance lies in making the signal dependent on that precise *alignment* between two different semantic representations, which is a very subtle way to achieve coherence.

Meng: I’m particularly interested in their ability to deploy this in a black-box API environment because we aren't needing custom software running on the user's side.

Lalam: It allows us to provide provenance without forcing the AI model into an unnatural or specific stylistic change, which is essential for maintaining natural linguistic expression.

Summary of Method: Tom: We’ve covered the basic concept and authorship, but now we want to dig into a summary of how Dual-Embedding Watermarking actually functions compared to the traditional methods.

Jane: The core idea is that instead of just adding fixed random noise, they are calculating biases based on how semantically related the context is to each potential next word.

Lu: This creates a very subtle and targeted signal, which is crucial for maintaining the natural flow of language because it’s not a rigid pattern.

Meng: I like that they are using this at the inference-time logit level; it’s a very clean integration point for engineers to understand where the AI is being influenced.

Lalam: This method allows us to see how we can give provenance while respecting the natural way language evolves, which is vital for maintaining trust in our outputs.

Tom: So, the authors take both an embedding of the entire preceding context and an embedding of each candidate token to create that targeted bias.

Jane: It’s essentially a dynamic assessment: "How well does this next word align with the overall meaning of what we've already written?" And that alignment dictates how much bias is applied to make it detectable.

Lu: This creates a very stable signal, which is achieved by leveraging the inherent mathematical properties of semantic space, making it robust against simple surface-level changes.

Meng: The fact they are doing this allows us to deploy it in a black-box setting because we are modifying the logit distribution directly during generation, not requiring any specific external code.

Lalam: It’s a way for us to ensure that our AI creations carry their history with them, acting as a subtle but reliable signature of authorship.

Improvements and Results: Tom: We’ve established the mechanism and its basic advantages; now we want to talk about the specific improvements this method makes over existing watermarking techniques.

Jane: The biggest win, according to the paper, is robustness against both paraphrasing—rewriting the text—and translation. Most prior methods failed if you just altered the language or rephrased it.

Lu: It’s not just surviving these attacks; it’s achieving consistent detection rates even when semantically invariant transformations occur, which is a huge theoretical leap forward from surface-level hashing schemes.

Meng: I also noticed that efficiency isn't a major drawback to worry about, which is fantastic for deployment because we don't have to add massive processing delays to the generation time.

Lalam: When considering real-world use, this means we can deploy watermarking without sacrificing the user experience or slowing down the pace of information flow for people using AI.

Tom: That’s a huge practical win, but let’s look at translation specifically—the results are striking when you look at the data.

Jane: It seems that because it is checking semantic alignment rather than exact word matches, the meaning carries the watermark across language barriers effectively.

Lu: This is where the concept of "semantic coherence" really pays off, as you’re seeing the signal degrade very slowly under those kinds semantic shifts.

Meng: The performance metrics are impressive; for instance, they see a true positive rate of up to sixty-five percent for German translation, which is significantly better than many other approaches.

Lalam: It means that the AI can generate content in different languages, and we still have a reliable way to trace that content back to its source using this method.

Conclusion: Tom: We’ve covered the core concept, how it works, why it's better, and what the results showed; let’s wrap up our discussion by summarizing the overall impact of Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings.

Jane: It provides a dependable tool that allows us to track AI-generated content while remaining highly efficient and resilient to modifications, which is a huge step toward peace of mind.

Lu: The theoretical groundwork is solid, ensuring that the signal remains stable even under conditions of semantic transformation, which is very satisfying to see.

Meng: Practically, it delivers a scalable solution that won't bog down high-throughput AI systems when we put it into production environments.

Lalam: It helps us build a more accountable and trustworthy future for AI interactions by providing clear provenance markers that everyone can rely on.

Tom: That’s a great way to summarize the impact of this work, combining stability with low overhead.

Lu: The whole research really shows that embedding-based methods are moving toward much more resilient solutions than the simple surface-level ones we used to see.

Meng: And the fact that it performs well across various languages confirms its practical utility in a real deployment scenario where language diversity is high.

Lalam: It allows us to continue trusting LLM outputs while ensuring that accountability remains intact for all users and stakeholders involved.

Tom: This whole discussion of Dual-Embedding Watermarking has been fascinating, so I think we’ll be seeing a lot more work like this in the future, and it's time for us to wrap up.

Department of Mathematics and Computer Science · Freie Universität Berlin

cs.CL, cs.CR

Submitted: 2026-06-30

Updated: 2026-09-04

Comments: Accepted to Findings of EMNLP 2026. 22 pages, 10 tables, 1 figure

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: The paper addresses the persistent challenges in attributing Large Language Model (LLM)-generated content, which is often rewritten or translated to meet specific demands.

Key concepts

Dual Semantic Embedding
The authors use two types of embeddings—one representing the overall context and another for each potential next word. By calculating biases based on how these two semantic representations align, the method creates a subtle signal that is inherently tied to meaning, rather than simple structural patterns.
Inference-time Logit Level Bias
This refers to the mechanism where biases are calculated at the exact point an AI model decides which token to output next. This allows for clean integration into a black-box API environment, as it modifies the logit distribution directly during generation without needing custom external software.
Semantic Alignment
The core idea is a dynamic assessment: checking how well a candidate word aligns with the overall meaning of the preceding text. This method creates a stable signal that remains detectable even when semantically invariant transformations, such as rephrasing or translation, occur.

Terminology

Summary

The paper addresses the persistent challenges in attributing Large Language Model (LLM)-generated content, which is often rewritten or translated to meet specific demands. Existing surface-level watermarking schemes are highly vulnerable to these semantic shifts, making reliable detection difficult. This work introduces Dual-Embedding Watermarking (DEW), a robust semantic watermarking scheme designed to enhance resilience against paraphrasing and translation by leveraging both contextual and token-level semantics, thereby providing a practical solution for safeguarding LLM outputs while maintaining competitive text quality.

How it works

DEW utilizes a signal processing methodology that applies algebraic vector-space operations to derive a watermark signal that degrades gracefully under semantic shifts. The core of the method lies in calculating the alignment between two distinct types of semantic embeddings: contextual and token-level. This process is designed to be robust, meaning that semantically similar tokens receive similar signals, which significantly improves robustness against translation.

Key Components of DEW

The system relies on several components to achieve its functionality:

  • Dual Embedding Models: The method incorporates two models: a token embedding model (M T) and a context embedding model (M C).

  • Secret Key and Projection: A cryptographically secure PRNG is seeded with a secret key K. This key is used to generate pseudo-random matrices (R T and R C) which obfuscate the embeddings.

  • Top-m Selection: At each generation step, the system selects the top m tokens with the highest logits as candidates for embedding.

The Dual-Embedding Advantage

Unlike prior methods that only condition on context semantics, DEW explicitly addresses inter-token semantic similarity. By computing separate embeddings for both the preceding context and each candidate token, DEW can assign similar biases to tokens that are semantically related. This dual approach ensures that the watermark signal is not easily removed by synonym substitution or translation. The process involves:

  • Obfuscating the embeddings through projection matrices (P T = E T R T).

  • Calculating the logit bias b via the dot product of projected context and projected token embeddings.

Watermark Insertion and Bias Calculation

The watermark is inserted by adding a calculated bias vector b to the original LLM logits (' = + b). This bias is determined by:

b = lambda times (gamma n times P T P C)

Where P T and P C are the projected token and context embeddings, respectively. The dot product of these two vectors serves as a key-based semantic alignment signal, quantifying the degree of alignment between them.

Experimental Performance and Efficiency

The experimental results demonstrate that DEW outperforms previous semantic baselines in several critical areas:

  • Robustness: It achieves superior performance against paraphrasing and translation. For instance, under German translation, DEW achieved up to 65.0% True Positive Rate (TPR) at a 1% False Positive Rate (FPR).

  • Efficiency: DEW maintains low computational overhead during both text generation and watermark detection, remaining one of the most efficient semantic watermarking schemes.

  • Text Quality: The method achieves competitive text quality, ensuring that the added biases do not significantly degrade the fluency or coherence of the generated text.

Improvements for AI systems

As a diligent researcher, I have analyzed the provided paper. While Dual-Embedding Watermarking (DEW) represents a significant advancement in robustness against paraphrasing and translation, its current implementation contains specific vulnerabilities and limitations that can be addressed to create a more resilient system.

The following improvements are highly specific and technical:

Current Vulnerability: The watermark signal relies on the cosine similarity between projected context (p C) and projected token embeddings (p T). This semantic alignment is predictable, making it susceptible to count-based watermark stealing attacks (as shown in Table 4).

Improvement: Implement a Dynamic Key Modulation (DKM) layer within the projection matrices R T and R C. Instead of using fixed, secret PRNG seeds for R T and R C, the key should be dynamically modulated by a hash of the immediate preceding token ID (Hash(x t-1)) in conjunction with a time-step counter.

R T(t) = PRNG(Key, t, x t-1)

Technical Specificity: This ensures that the projection matrices are not static across tokens or even across different generations, preventing an attacker from training a single consistent logit reweighting model.

Current Vulnerability: The performance of DEW is highly sensitive to the specific embedding models used (e.g., Llama-3 vs. Gemma-7B), indicating that the inherent structure of the LLM's latent space dictates robustness, not just the DEW mechanism itself.

Current Vulnerability: The detection relies on comparing the document-level score against a fixed analytic threshold tau a, derived from the Gaussian approximation Z L about N(0, 1). This assumes independence and consistency in token correlations.

Current Vulnerability: While DEW is efficient compared to semantic baselines, its complexity remains O(m times d T times n) per step. The current implementation relies on a general PRNG for R T and R C.

The refined DEW system will achieve:

  1. Unmatched Cross-Lingual Consistency: By coupling token and context semantics with dynamic projection, it maintains a robust signal even when text is translated into languages where semantic structures differ significantly from the original training distribution (e.g., achieving 70%+ detection rates on highly complex translations).

  2. Guaranteed Signal Integrity: The system will be virtually immune to drift or statistical degradation caused by local semantic changes, as the MLNP and BOC ensure that the signal integrity is preserved across both minor lexical edits and major semantic shifts.

  3. Resilience Against Adversarial Attacks: The DKM layer makes it nearly impossible for an attacker to train a single model to bypass detection, as the watermark signal is no longer static or easily predictable based on fixed context windows.

  4. Operational Reliability: The ACAT mechanism ensures that the system's ability to distinguish watermarked text from human-authored text remains consistent across all operational environments, regardless of local statistical anomalies in a highly-correlated sequence.

Abstract

This work presents Dual-Embedding Watermarking (DEW), a semantic watermarking scheme for large language models (LLMs) that leverages contextual and token-level embeddings to enhance robustness against paraphrasing and translation. DEW utilizes a signal-processing methodology, applying algebraic vector-space operations to token and context embeddings to derive a watermark signal that degrades gracefully under semantic shifts. The method obfuscates the watermark by projecting embedding vectors through pseudo-random matrices seeded with a secret key. Experimental results show that dual-embedding watermarking can offer state-of-the-art robustness, particularly against translation, while incurring relatively low computational overhead compared with other semantic schemes. At lower watermark strength, DEW also maintains competitive text quality, suggesting that dual-embedding signals provide a promising substrate for robust semantic watermarking.

Sources

Related papers