PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

arXiv:2608.29624 · cs.CL · Submitted 2026-08-30 · Read on arXiv

cs.CL

Submitted: 2026-08-30

Updated: 2026-08-30

Code: https://github.com/sjmeis/PrivBench

Terminology

• Differentially Private — 33× in this paper · explained in 3 episodes
• Natural Language Processing — 32× in this paper
• Differential Privacy — 27× in this paper · explained in 4 episodes
• Benchmark — 19× in this paper · explained in 2 episodes
• Benchmarking — 17× in this paper · explained in 4 episodes
• Transformer — 9× in this paper · explained in 2 episodes
• Fine-Tuning — 7× in this paper · explained in 5 episodes
• Federated Learning — 6× in this paper · explained in 16 episodes
• Adam — 5× in this paper · explained in 2 episodes
• Embeddings — 5× in this paper · explained in 2 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Perplexity — 4× in this paper · explained in 3 episodes
• Distillation — 3× in this paper · explained in 2 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Consistency — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• F1 Score — 2× in this paper · explained in 3 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• In-Context Learning — 2× in this paper · explained in 4 episodes
• Knowledge Distillation — 2× in this paper · explained in 11 episodes
• Representation Learning — 2× in this paper · explained in 2 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Semantic Similarity — 2× in this paper · explained in 2 episodes
• AUC — 1× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Attack Success Rate (ASR) — 1× in this paper · explained in 2 episodes
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• False Positive Rate — 1× in this paper
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• Personalization — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Sparse Attention — 1× in this paper · explained in 5 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes
• True Positive Rate — 1× in this paper
• Workshop on Privacy in Large Language Models — 1× in this paper

Sources

• SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Data
• On the Privacy Risk of In-context Learning
• Submix: Practical Private Prediction for Large-Scale Language Models
• A General Pseudonymization Framework for Cloud-Based LLMs: Replacing Privacy Information in Controlled Text Generation
• Textwash -- automated open-source text anonymisation
• Split-and-Denoise: Protect large language model inference with local differential privacy
• Differentially Private Decoding in Large Language Models
• Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey
• SynBench: A Benchmark for Differentially Private Text Generation

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering