Black-Box Forensics for Conversational LLM Agents

The gist: Black-box forensics for conversational LLM agents offers a path to accountability for systems hidden behind anonymous endpoints by identifying the base model and system prompt purely through conversation.

Attribution and Fingerprinting Capabilities

The paper introduces two complementary capabilities for black-box forensics: attribution, which identifies the base model or system prompt from a closed set of options, and fingerprinting, which detects whether two endpoints share the exact same—possibly never-beforeseen—system prompt. Attribution classifiers identify the base model behind an agent with 98% accuracy from a few turns of nonadversarial conversation <ref:2606.22698#pg5> The cross-encoder fingerprinting method achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts, and aggregating 50 interaction conversations from each target agent boosts AUC to 0.943 <ref:2606.22698#pg5>

Methodology and Active Elicitation

The framework employs an active elicitation paradigm where a detective agent steers the conversation with the target agent to control the topic and isolate the target’s structural fingerprint from semantic noise <ref:2606.22698#pg4> The study defines a conversational LLM agent (m, p) as being parametrized by its base model m and system prompt p <ref:2606.22698#pg5>

Attribution Techniques

For attribution of base models, the researchers apply techniques inspired by authorship attribution methodologies to the multi-turn paradigm <ref:2606.22698#pg5> They implement two standard classifier families: a sparse baseline using unigram and character-level TF–IDF alongside stylometric features, and a modern paradigm utilizing language models as dense classifiers by fine-tuning QWEN-4B-INSTRUCT with LoRA adapters <ref:2606.22698#pg5> Performance is heavily gated by the base model’s inherent responsiveness to instructions, with models like GPT-4.1-NANO yielding pairwise accuracies above 90% <ref:2606.22698#pg6>

Fingerprinting Techniques

To determine if two conversations originated from the same criminal organization, the approach involves training a model to detect when two conversations share the same system prompt <ref:2606.22698#pg5> The cross-encoder method leverages cross-attention using ELECTRA-large and BERT-base to encode the conversations and output log probabilities for classifying same versus different, employing a cross-entropy loss to update the encoder This method achieves 0.768 AUC and an F1 of 0.703 Aggregating from 50 conversations per target raises performance to an AUC of 0.943 and an F1 of 0.79

Evaluation and Robustness

The experiments utilize a synthetic corpus of 240k labeled transcripts generated by leveraging the detective LLM agent (powered by QWEN-4B-INSTRUCT) to conduct standardized interactions with target agents Evaluation splits involve matching conversations based on topic and creating an even number of pairs with the same agent and with a different agent in both splits Robustness checks investigate performance against topic shifts, unseen detective agents, sampling parameters like temperature and max tokens, context lengths, and paraphrase attacks The cross-encoder method shows robustness to changes in topic, sampling parameters, punctuation, and the detective model with AUC drops of less than 0.03 However, context length significantly impacts performance, with a one-turn conversation causing a steeper 0.20 decrease in AUC

Conclusion and Deployment Recommendations

The work introduces techniques for black-box forensics, specifically attribution of base models and system prompts and fingerprinting of system prompts on known base models Practitioners are recommended to use attribution in conjunction with fingerprinting, as fingerprinting can be used to group conversations together, and multiple conversations from different endpoints can be used to decrease uncertainty Once a specific suspicious set of outputs is linked to one another, the attribution techniques can be leveraged to trace these outputs to the specific model provider Future work should include integrating this system into real-world workflows, such as a honeypot LLM system, designed to entrap scammers and use this information to trace cyber criminals defrauding the globe

Limitations and Ethical Considerations

Utilizing a meticulously controlled synthetic corpus is a deliberate choice that bypasses significant ethical and legal gray areas regarding prompt extraction from proprietary systems To mitigate surveillance risks, real-world deployments should adhere to operational protocols including pre-flight target validation, data minimization and automated sanitization, context-aware abort mechanisms, and mandatory auditability These protocols ensure that when cross-encoder models successfully evaluate conversational pairs under entirely unseen system prompts, they are mapping prompt-driven behavioral blueprints rather than topic-driven semantic shifts

System Prompts Examples

The paper provides a set of curated system prompts exhibiting distinct behavioral profiles across various categories such as Professional, Warm and Supportive, Friendly and Conversational, Concise, Thorough and Explanatory, Socratic Guide, Educational, Creative, Analytical, Empathetic, Cheerful and Upbeat assistant These prompts cover 13 distinct categories of conversational styles The topics studied span 55 different scenarios ranging from booking a flight through an airline live-chat agent to persuading a friend over messaging app to donate to a disaster-relief charity This comprehensive set of prompts is used for training and evaluating the forensic methods described in the paper The data includes 70 seed topics spanning customer support as well as adjacent dialogue settings such as negotiation and interpersonal communication The dataset summary for zero-shot fingerprinting of system prompts shows that same pairs are those that have the same system prompt and base model (m, p), and different pairs have the same base model but different system prompts: (m, pi) and (m, pj), where pi!= pj The dataset statistics for zero-shot fingerprinting of system prompts include a total of 31,500 same pairs and 44,100 different pairs The performance comparison on conversations between QWEN-4B-INSTRUCT detective agent and GPT-4.1-NANO target base model shows that the all three turns result in an AUC of 0.871 The cross-encoder and bi-encoder methods perform significantly better than baseline methods across all subsets This work is the first to study the attribution of system prompts in a conversational black-box setting, where there is no access to model weights or system prompt internals The paper's key contributions include attribution of base models and system prompts from a few turns of nonadversarial conversation with 98% accuracy The cross-encoder achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts These capabilities serve everyday platform governance by revealing silent drifts in backend checkpoints, safety layers, or system prompts The paper's methods require no perturbation of the base model, no access to its output logits, and no ground-truth system prompts at training time Our key contributions include attribution of base models and system prompts from a few turns of nonadversarial conversation with 98% accuracy The cross-encoder achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts The paper's methods require no perturbation of the base model, no access to its output logits, and no ground-truth system prompts at training time Our key contributions include attribution of base models and system prompts from a few turns of nonadversarial conversation with 98% accuracy The cross-encoder achieves an AUC of 0.768 and an F1 of 0.

1. Identification of base models using non-adversarial interaction: Attribution of base models and system prompts is effective in binary classification setting as well, but it is dependent on (1) the base model and (2) the semantic similarity of system prompts. This allows investigators to trace AI-enabled scams back to the providers whose models power them with 98% accuracy from a few turns of nonadversarial conversation.

2. Detection of silent API changes via fingerprinting: The cross-encoder method allows for detecting if two endpoints share the same hidden configuration, even if unseen: our cross-encoder achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts. This capability enables auditors to detect silent drift—without any knowledge of what the configuration was or what it became.

3. Scaling fingerprinting for large-scale monitoring: Aggregating conversations enhances detection accuracy: aggregating 50 interaction conversations from each target agent boosts AUC to 0.943 and an F1 of 0.77. This allows platforms to cluster distinct scam campaigns and detect changes with high robustness across a larger sample size.

4. Model-specific vulnerability profiling: Attribution helps tailor security responses: attribution tells defenders which model they are actually facing. This enables red-teaming to be done with precision, as jailbreaks transfer only partially across models and prompt templates.

5. Robustness against adversarial perturbations: The method demonstrates resilience against common evasion techniques: changes in topic, sampling, punctuation, and the detective model result in AUC drops of less than 0.03. This ensures forensic accuracy even when dealing with varied conversational styles or prompt obfuscation.

As LLM-powered scams proliferate, black-box forensics for conversational LLM agents offers a path to accountability for systems hidden behind anonymous endpoints. Identifying the base model behind a chatbot endpoint (attribution), without model parameter access or knowledge of the hidden system prompt, would let investigators trace AI-enabled scams back to the providers whose models power them. Detecting when two endpoints run the exact same system prompt (fingerprinting), even one novel and unseen, would link individual scams into criminal networks and expose silent API changes. We conduct an empirical investigation of both capabilities. Our attribution classifiers identify the base model behind an agent with 98% accuracy from a few turns of non-adversarial conversation. Attribution of system prompts, while possible, requires retraining on a large amount of data for each prompt; system prompts in the wild are unbounded and ever-changing, making this approach costly. To tackle this more open-ended setting, our cross-encoder fingerprinting method achieves an AUC of 0.768 and an F1 of 0.703 on entirely unseen system prompts, and aggregating 50 interaction conversations from each target agent boosts AUC to 0.943. Conversational agents with unseen system prompts can thus be fingerprinted with robust accuracy from a few turns of ordinary conversation.