EVIL-Detect for NLPCC 2026 Shared Task 6: LLM-Generated Text Detection

This paper presents EVIL-Detect, a multi-signal ensemble framework with conflict-aware fusion for NLPCC 2026 Shared Task 6, which requires three-class classification of Chinese text into human-written text (HWT), LLM-generated text (LGT), and LLM-refined text (HLT). The system integrates edit-extent regression, zero-shot likelihood-contrast signals, lexical statistics, and conservative text rules. With calibrated decision boundaries and conflict-aware integration, the system improves robustness under strong out-of-distribution shifts, achieving a macro-F1 score of 0.8888 and ranking first in the official evaluation. The code is available at https://github.com/bbbbhrrrr/evildetect.

The paper motivates the work by noting that most existing detection methods focus on English binary detection, while Chinese detection is less explored and cannot be handled by direct transfer from English due to differences in tokenization, lexical granularity, and writing conventions. NLPCC 2026 Shared Task 6 extends the task to three classes, reflecting real AI-assisted writing scenarios where users often use LLMs to rewrite or polish human-written text.

The proposed EVIL-Detect system combines four types of evidence: supervised edit-extent signals from EditLens and Soft-EditLens, zero-shot LGT-tendency signals from EchoPrompt, lexical frequency statistics, and conservative text rules. The fusion module integrates these heterogeneous signals with conflict-aware decision logic and outputs the final HWT, LGT, or HLT prediction.

The supervised training module is inspired by EditLens, which formulates AI-edited text detection as continuous edit-extent estimation rather than pure discrete classification. In this setting, each training group contains a triplet (h, g, t), where h, g, and t denote the HWT, LGT, and HLT versions respectively. The editing-extent target is defined on the HWT–HLT–LGT axis using HWT and LGT as two anchors: r(h) = 0, r(g) = 1, r(t) = d(h, t), where d(h, t) ∈ [0, 1] measures how far the LLM-refined text t moves away from the corresponding human-written source h. This formulation treats HLT as an instance-dependent soft editing state rather than an independent hard class.

Two variants of the supervised module are instantiated. EditLens uses a surface-level soft character n-gram distance to construct d(h, t), with the best-ranked configuration having internal sweep id 044, using orders 1–7 with linearly increasing weights. Soft-EditLens replaces surface n-gram overlap with semantic phrase-level soft matching, and also trains an ordinal bucket predictor that discretizes the editing-extent axis into ordered intervals.

The zero-shot detection module uses EchoPrompt, a training-free prompting module that computes a normalized likelihood-contrast score using a prompted instruction-tuned model and its corresponding base model. Multiple fixed prompt/model branches are used to reduce dependence on a single prompt or model.

The lexical frequency statistics module builds label-wise frequency lexicons from the training set and retains discriminative n-grams. It computes several derived statistics, including LGT-vs-HWT tendency, AI-vs-HWT tendency where LGT and HLT are merged as AI-assisted text, the fraction of machine-oriented lexical units, and the fraction of HWT-oriented lexical units.

The fusion module integrates the outputs of the preceding components. The starting point is the EditLens editing-extent score, which is converted into a base label using two calibrated boundary settings. These base labels are combined with Soft-EditLens regression/bucket signals and a panel of nine binary LGT-support votes from zero-shot, edit-based, and lexical views. If the two base labels agree, the agreed label is kept directly. If they disagree, a conflict-aware resolver R(·) is used, with validation-calibrated thresholds. For HWT/LGT conflicts, R outputs HWT when the Soft-EditLens regression score zr(x) ≤ γH, otherwise outputs LGT when the number of active LGT-support votes v(x) ≥ γHL, and otherwise falls back to HLT. For HWT/HLT conflicts, R outputs HLT iff the bucket index zb(x) ≥ βT; otherwise it outputs HWT. For HLT/LGT conflicts, R outputs LGT iff v(x) ≥ γTL; otherwise it keeps HLT.

After conflict-aware integration, a small set of conservative text rules is applied as final corrections. Raw HTML-like or XML-like structural markup, such as,,,

, and, is treated as strong evidence for LGT. Script or rendering residues, such as document.write("

") or unfinished tag sequences, also trigger an LGT correction. Explicit rewriting or polishing traces support HLT when the fused label is not already LGT.

The experimental setup uses the official data released for NLPCC 2026 Shared Task 6. The training set is sampled and adapted from the CUDRT dataset and covers four generators (GPT-4, Qwen-family generators, ChatGLM, and Baichuan) across two domains (news and academic writing). The two hidden test phases, testp1 and testp2, are drawn from the Chinese split of DetectRL-X and are constructed as out-of-distribution data. The cleaned training set has 57,589 samples, testp1 has 3,600 samples, and testp2 has 1,152 samples.

The module configuration uses EditLens with Qwen3.5-4B-Base + LoRA for editing-extent label, Soft-EditLens with Qwen3.5-9B-Base + LoRA for regression/bucket scores, EchoPrompt with Qwen3.5-4B and Qwen2.5-1.5B for the LGT vote panel, lexical frequency with label-wise n-gram lexicon for the LGT vote panel, and a deterministic fusion module.

The main results show that EVIL-Detect achieves macro-F1 scores of 0.8913 on testp1 and 0.8888 on testp2. LGT obtains the highest F1 scores, reaching 0.9267 on testp1 and 0.9219 on testp2. HLT is consistently the most challenging class, with F1 scores of 0.8391 and 0.8407, consistent with its intermediate nature between HWT and LGT.

Component analysis on testp1 shows that EditLens with the rank044 configuration is the strongest standalone component, reaching 0.8494 macro-F1, outperforming the initial soft-n-gram configuration (0.8289). The best zero-shot strategy reaches 0.6325 macro-F1, and the best lexical classifier reaches 0.5535 macro-F1. With conflict-aware fusion, EVIL-Detect reaches 0.8913 macro-F1, improving over EditLens by 4.19 points.

Representative alternative designs evaluated on testp1 include Binoculars (0.3928 macro-F1), generative SFT with GLM-4-9B (0.1896 macro-F1), and generative SFT with Qwen3.5-9B (0.1690 macro-F1), showing strong class bias and supporting the design choice of using editing-extent modeling as the base signal.

The ablation study on testp2 shows that the EditLens-only baseline obtains 0.8411 macro-F1. Adding conflict-aware integration and high-precision text rules improves the score to 0.8816, and the full system reaches 0.8888 after combining two calibrated EditLens decisions. The improvement mainly comes from boundary cases: compared with EditLens only, the full system reduces LGT→HLT errors from 59 to 30 and HLT→LGT errors from 58 to 28.

The paper concludes that robust three-class detection cannot rely on a single signal; instead, edit-strength modeling, likelihood-based evidence, lexical cues, and high-precision corrections provide complementary information for distinguishing HWT, LGT, and HLT. The work is supported by the Postdoctoral Fellowship Program of China Postdoctoral Science Foundation under Grant Number GZC20251076.

Improvements to AI systems:

1. Multi-signal ensemble with conflict-aware fusion: Instead of relying on a single detection signal, AI systems can integrate heterogeneous evidence (edit-extent regression, likelihood-contrast scores, lexical statistics, and rule-based corrections) with explicit conflict-resolution logic. This improves robustness under out-of-distribution shifts, as demonstrated by a 4.19-point macro-F1 gain over the best single component.

2. Continuous edit-extent modeling for three-class detection: Replace hard discrete classification with a regression-based editing-extent axis (0 for human-written, 1 for LLM-generated, intermediate values for refined text). This treats LLM-refined as a soft, instance-dependent state rather than an independent class, improving boundary-case accuracy (reducing LGT→HLT errors from 59 to 30 and HLT→LGT errors from 58 to 28).

3. Calibrated decision boundaries with validation-based thresholds: Use two calibrated boundary settings for the base edit-extent score and a conflict-aware resolver with validation-tuned thresholds (e.g., γH, γHL, βT, γTL). This allows the system to adaptively decide between conflicting signals, reducing misclassifications in ambiguous cases.

4. Zero-shot likelihood-contrast voting panel: Use multiple fixed prompt/model branches (e.g., Qwen3.5-4B and Qwen2.5-1.5B) to compute normalized likelihood-contrast scores, reducing dependence on a single prompt or model. This provides complementary evidence that improves recall for LLM-generated text (LGT F1 reaches 0.9267).

5. Lexical frequency statistics with discriminative n-grams: Build label-wise frequency lexicons and derive derived statistics (LGT-vs-HWT tendency, AI-vs-HWT tendency, machine-oriented lexical fraction). This adds a lightweight, interpretable signal that captures surface-level stylistic differences, improving performance when combined with deeper models.

6. Conservative high-precision text rules as final corrections: Apply deterministic rules for strong evidence (e.g., raw HTML/XML markup, script residues, explicit rewriting traces) to correct edge cases without harming precision. This improves overall macro-F1 by 0.7 points on testp2 when added to the fused model.

7. Semantic phrase-level soft matching (Soft-EditLens): Replace surface n-gram overlap with semantic phrase-level matching and ordinal bucket prediction to better capture editing extent, especially for paraphrased or reworded text where surface overlap fails.

8. Instance-dependent anchor-based target design: Define editing-extent targets using human-written and LLM-generated versions as anchors (r(h)=0, r(g)=1, r(t)=d(h,t)), allowing the model to learn a continuous scale that generalizes across different generators and domains.

What the improved AI system can do:

• Accurately classify Chinese text into human-written, LLM-generated, or LLM-refined categories, even under strong out-of-distribution conditions (macro-F1 0.8888 on unseen test sets).

• Handle real-world AI-assisted writing scenarios where users polish or rewrite human text, by modeling the degree of editing rather than forcing a binary decision.

• Provide robust detection across multiple LLM generators (GPT-4, Qwen, ChatGLM, Baichuan) and domains (news, academic writing) without retraining.

• Reduce false positives and false negatives in boundary cases (e.g., lightly edited text vs. fully generated text) through conflict-aware fusion and calibrated thresholds.

• Operate with a mix of supervised and zero-shot components, making it adaptable to new languages or domains with minimal labeled data.

• Offer interpretable outputs by combining lexical statistics and rule-based corrections with deep model scores, enabling auditability and debugging.

The rapid development of large language models (LLMs) has increased the need for reliable detection of LLM-generated text, especially in realistic Chinese scenarios involving human-written text (HWT), LLM-generated text (LGT), and LLM-refined text (HLT). This paper presents EVIL-Detect, a multi-signal ensemble framework with conflict-aware fusion for NLPCC 2026 Shared Task 6. The system integrates edit-extent regression, zero-shot likelihood-contrast signals, lexical statistics, and conservative text rules. With calibrated decision boundaries and conflict-aware integration, our system improves robustness under strong out-of-distribution shifts, achieving a macro-F1 score of 0.8888 and ranking first in the official evaluation. Our code is available at https://github.com/bbbbhrrrr/evildetect.