On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence

Named-entity recognition (NER) on-device deployment requires evaluating models based on deployability factors like latency and reliability, rather than just leaderboard accuracy. This study investigates nine systems spanning classical taggers, bidirectional encoders, and generative LLMs across three datasets to map the accuracy, cost, and reliability frontier for NER systems intended for local execution.

The gist

Encoders are the deployable choice [because] they match or trail by a little at one-ninth to one-twentyfourth the size, at millisecond-to-second latency, and with zero malformed output.

System Comparison and Accuracy Frontier

The study places nine systems—a classical tagger (spaCy), bidirectional encoder specialists (GLiNER), and local generative LLMs (Qwen3, DeepSeek-R1)—on a frontier defined by accuracy, latency, and output validity. The results show that Encoders lead where text is noisy and entities are novel, while the fair mid-size LLM is competitive on clean text. Specifically, Qwen3-4B-Instruct leads on clean CoNLL (0.753) and RSS-News (0.684), but encoders outperform it when noise or novelty is present, such as GLiNER-large topping WNUT with 0.599 compared to the 4 B model's 0.506. The paper notes that Encoders beat LLMs is not the honest headline; rather, the honest headline is that encoders are the deployable choice, which the next axes establish.

Annotation-Free Evaluation Protocol

The researchers developed an evaluation protocol to assess performance without a human-annotation budget. This involved building a silver gold from a cross-family LLM judge panel, which measured fidelity against benchmark human gold and then against a full human re-validation of the corpus itself (strict F1 0.95, an upper bound). A key finding is that gold provenance flips the paradigm ranking: moving from LLM-authored silver to human gold raises every encoder and lowers every generative model. This flip is observed in Table 7, where GLiNER-large overtakes R1-8B on human gold.

Output Validity and Generative Model Pathology

A critical metric introduced is output validity, which measures whether a generative system fails to terminate a schema-valid object within its budget. The study demonstrates that the invalid-output failure of small generative models is fixed by scale, not by output budget. For instance, Qwen3-0.6B emits 27% invalid output on long inputs (4000 tokens), whereas Qwen3-4B-Instruct shows 0% invalid under either budget. This pathology is shown to be fixed by scale, not by output budget, confirming that the failure is non-termination, not malformed decoding.

Confidence Calibration and Reliability Tools

The study characterizes the per-span confidence of GLiNER encoders, noting that while they rank correctness well (AUROC 0.76 to 0.86), they are badly overconfident (ECE 0.24 to 0.47). The authors show that temperature scaling roughly halves ECE, and that confidence thresholding provides a small honest out-of-sample F1 gain. Furthermore, an all-local small→large cascade is shown to yield a modest, corpus-dependent gain over random routing. The paper concludes that the confidence tracks correctness but not novelty.

Deployability Strategy: Selective Prediction and Cascades

The reliability analysis leads to practical deployment strategies. Selective prediction trades coverage for risk, where thresholding confidence helps, yielding gains like +0.133 for GLiNER-medium on WNUT. The final proposed strategy is an all-local, encoder→encoder, span-gated NER configuration in the gap between them. This cascade routes uncertain documents to a local stronger tier rather than a cloud LLM as seen in LinkNER. The paper concludes that GLiNER-small the efficiency default and advises practitioners to Rank, do not read, the confidence, using it instead as an all-local cascade.

Limitations and Caveats

The characterization of difficulty is caveated because static metrics like entity density or unseen-entity ratio do not predict NER failure at the desired resolution. The effect that survives is only the coarse between-dataset novelty property. Additionally, the fidelity bound relies on a judge panel whose gold was seeded from silver, meaning it is an upper bound and LLM-authored gold systematically flattens LLM-style systems. The study also notes that confidence analysis only covers the GLiNER family.

Based on the scientific paper provided, here are specific improvements for AI systems, categorized by paradigm:

) Improvements for On-Device Deployment (Focusing on Encoders):

1. The core recommendation is to deploy a compact bidirectional encoder (like GLiNER) rather than relying solely on large generative LLMs for NER.

2. An improved system should utilize an all-local, encoder→encoder cascade architecture: use a small base model (e.g., GLiNER-small) and route documents to a larger local tier (GLiNER-large) only when the initial confidence is low.

3. This cascade can be implemented using confidence-gated routing, where the decision to escalate is based on the model's native per-span confidence score, rather than relying on external, potentially miscalibrated metrics.

) Improvements for Generative LLMs (Focusing on Qwen/DeepSeek):

1. Generative models should be used primarily when accuracy alone is paramount (e.g., on clean newswire where they are competitive), but their deployment must account for a significant reliability risk: they emit up to 27% invalid output on long inputs, which is a failure of non-termination/structure, not just bad tokens.

2. To mitigate the invalid output pathology, generative systems must be constrained by strict structured output decoding (JSON schema) and a fixed token budget. The system should be designed to fail gracefully (truncate the object) rather than producing malformed spans that break downstream processing.

3. When generating NER, reasoning modes should be disabled, as they show no benefit over smaller instruction-tuned models for extraction tasks.

) Improvements for Evaluation and Trust (Focusing on Methodology):

1. Instead of relying solely on leaderboard F1 scores, the evaluation protocol must incorporate a measured fidelity bound. This involves using a cross-family judge panel (e.g., Gemini-Flash-Lite, Nemotron-3-Ultra) to create an annotation-free silver gold set.

2. This silver gold should then be rigorously validated against a full human re-validation of the target corpus to establish a strict fidelity metric (e.g., 0.95 F1 bound).

3. System ranking comparisons must be anchored on this human gold provenance, as LLM-authored silver gold is known to systematically flatter LLM-style systems.

) Improvements for Confidence and Decision Making (Focusing on Robustness):

1. The native per-span confidence scores from encoders should not be treated as direct probabilities; they should be used specifically for ordering spans, abstention, and routing decisions.

2. To turn the miscalibrated confidence into a useful tool, a post-hoc fix like temperature scaling (fitted via NLL on inverted confidence) should be applied to generate calibrated probabilities where true probability is required.

3. For deployment, the system should implement selective prediction: only output spans above a carefully selected threshold to maximize F1 gain while managing risk, rather than keeping all predictions.