AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers

AWED-FiNER is an open-source collection of an agentic tool, a web application, and 53 state-of-the-art expert models that provide Fine-grained Named Entity Recognition (FgNER) solutions across 36 languages spoken by more than 6.6 billion people. The agentic tool enables routing multilingual text to specialized expert models to fetch FgNER annotations within seconds. The web-based platform provides a ready-to-use FgNER annotation service for non-technical users. Moreover, the collection of language-specific extremely small open-source state-of-the-art expert models facilitates offline deployment in resource-constrained scenarios, including edge devices. AWED-FiNER covers languages spoken by over 6.6 billion people, ranging from global languages like English, Chinese, Spanish, and Hindi, to low-resource languages like Assamese, Santali, and Odia, along with a specific focus on extremely low-resource vulnerable languages such as Bodo, Manipuri, Bishnupriya, and Mizo.

The expert detector models are fine-tuned models based on the SampurNER, CLASSER, MultiCoNER2, FewNERD, FiNERVINER, FiNE-MiBBiC, and APTFiNER datasets. These models integrate into the agentic tool AWED-FiNER, allowing seamless one-line integration into modern generative AI workflows. To the best of the authors' knowledge, this is the first comprehensive contribution covering an agentic tool, an interactive web app, and a collection of 53 expert models across 36 languages, serving 6.6 billion speakers, for the fine-grained named entity recognition task.

The experimental setup involved fine-tuning IndicBERTv2 (IndicBERTv2-MLM-Sam-TLM), MuRIL (muril-large-cased), and XLM-RoBERTa (XLM-RoBERTa-large) for FgNER using Hugging Face Transformers. Models were trained for six epochs with batch size 64 using AdamW (learning rate 5e-5, weight decay 0.01) on an NVIDIA A100 GPU. Performance was evaluated using SeqEval metrics, with best models selected by Macro-F1-score. For each language, when fine-tuned expert models are available across multiple datasets, the model with the highest Macro-F1 score is preferred for the AWED-FiNER Web Application and Agentic Toolkit. However, all 53 expert models are made available as an open source collection and can be used based on the purpose and the requirement for entity-type granularity.

The AWED-FiNER Suite consists of an Agentic Toolkit, an Interactive Web Application, and a collection of 53 small-sized expert models for offline deployment, including on edge devices. The Multilingual Expert Model Collection is a centralized repository of 53 fine-tuned expert models specialized for FgNER, which incorporates established benchmark datasets: FewNERD, MultiCoNER2, CLASSER, SampurNER, FiNERVINER, APTFiNER, and FiNE-MiBBiC. These multilingual expert models target a global audience of 6.6 billion speakers by leveraging multilingual encoder models: IndicBERTv2, MuRIL, and XLM-RoBERTa. As each of these compact models has fewer than 355 million parameters, offline deployment is facilitated in resource-constrained scenarios while addressing linguistic disparity across a range of languages, from widely spoken global languages to extremely low-resource and vulnerable languages.

The Interactive Web Application, the AWED-FiNER Hugging Face Space, serves as the backbone for both the interactive web application and the agentic toolkit. The interactive application, developed using the Gradio framework, enables real-time FgNER visualization across 36 languages. This infrastructure functions as a routing layer that dynamically selects and loads the appropriate compact expert models to minimize resource usage. By providing a unified portal, the space acts as a deployment benchmark for evaluating multilingual performance on complex FgNER taxonomies in resource-constrained environments.

The Agentic Toolkit utilizes this space as a callable tool, implementing the smolagents framework to enable automated workflows. It leverages structured metadata for the 36 supported languages and their fine-grained taxonomies to allow for seamless integration and automatic invocation during inference. This design bridges the gap between static model repositories and autonomous AI applications, ensuring that specialized entity recognition can be effectively integrated into Large Language Models and automated processing pipelines.

The conclusion states that AWED-FiNER delivers a scalable, efficient FgNER suite for 36 languages and 6.6 billion speakers, with strong support for low-resource and vulnerable languages. Its lightweight expert models and agentic tool enable fast, low-memory deployment in resource-constrained environments. To the best of the authors' knowledge, this is the first unified agentic tool, web application, and expert model collection for FgNER task serving 6.6 billion people.

The ethical statement emphasizes the preservation of low-resource vulnerable languages (Bodo, Manipuri, Bishnupriya, and Mizo) to prevent digital linguistic extinction. The datasets used for fine-tuning the models are released under MIT license, CC-BY-4.0, and CC0 licenses. The authors did not modify these datasets to correct for potential biases, and they used them as-is. They have cited all the sources of resources, tools, packages, and models used in this work. The AWED-FiNER Web Application, Agentic Tool, and the collection of 53 expert models are released under MIT license.

Based on the paper, I can implement the following specific improvements to AI systems:

Improvement: Build an agentic routing layer that detects the input language and automatically selects the best-performing expert model from the 53 available models (e.g., MuRIL for Assamese, XLM-RoBERTa for English, IndicBERTv2 for Gujarati).

Capability: The system can now process text in 36 languages—including extremely low-resource languages like Bodo, Manipuri, Bishnupriya, and Mizo—and return fine-grained entity annotations (e.g., person, organization, location, product, event) within seconds, without requiring the user to manually specify the language or model.

Improvement: Integrate the compact expert models (all under 355M parameters) into a quantized, ONNX-optimized inference pipeline for offline use.

Capability: The improved system can run on devices with limited memory (e.g., mobile phones, Raspberry Pi, or low-end servers) and still achieve state-of-the-art Macro-F1 scores (e.g., 85.83 for Italian, 84.64 for Swedish, 81.07 for Mizo) without internet connectivity, enabling NER in disaster relief, field research, or privacy-sensitive environments.

Improvement: Create a unified entity-type mapping layer that translates between the different taxonomies used in FewNERD, MultiCoNER2, CLASSER, SampurNER, FiNERVINER, APTFiNER, and FiNE-MiBBiC.

Capability: The improved system can accept text and return annotations in a consistent, user-defined fine-grained schema (e.g., 66 entity types from FewNERD or 34 from MultiCoNER2) regardless of which underlying expert model is invoked, making it easier to integrate into downstream IR and knowledge-base construction pipelines without retraining.

Improvement: Implement a fallback mechanism that, for any input language not in the 36 covered, uses script similarity and cross-lingual transfer (based on CLASSER's approach) to route to the closest available expert model.

Capability: The system can now provide reasonable fine-grained NER for related languages (e.g., Chittagonian via Bengali models, or Maithili via Hindi models) with minimal performance degradation, expanding effective coverage beyond 6.6 billion speakers.

Improvement: Deploy the Gradio-based web application as a callable API endpoint that supports batch processing and streaming input.

Capability: Non-technical users (e.g., journalists, social scientists) can paste or upload documents in any of the 36 languages and receive color-coded, fine-grained entity annotations in real time, with the system automatically selecting the best expert model per language—no coding or model selection required.

Improvement: Add a dedicated mode that prioritizes the four vulnerable languages (Bodo, Manipuri, Bishnupriya, Mizo) by loading their specific expert models first and providing confidence scores.

Capability: The improved system can assist linguists and archivists in automatically tagging endangered-language texts with fine-grained entities (e.g., person, location, organization) at F1 scores of 62.97–81.07, accelerating documentation and preservation efforts before these languages face digital extinction.

Named Entity Recognition (NER) is a foundational task in Natural Language Processing (NLP) and Information Retrieval (IR), which facilitates semantic search and structured data extraction. We introduce AWED-FiNER, an open-source collection of agentic tool, web application, and 53 state-of-the-art expert models that provide Fine-grained Named Entity Recognition (FgNER) solutions across 36 languages spoken by more than 6.6 billion people. The agentic tool enables routing multilingual text to specialized expert models to fetch FgNER annotations within seconds. The web-based platform provides a ready-to-use FgNER annotation service for non-technical users. Moreover, the collection of language-specific extremely small open-source state-of-the-art expert models facilitates offline deployment in resource-constrained scenarios, including edge devices. AWED-FiNER covers languages spoken by over 6.6 billion people, ranging from global languages like English, Chinese, Spanish, and Hindi, to low-resource languages like Assamese, Santali, and Odia, along with a specific focus on extremely low-resource vulnerable languages such as Bodo, Manipuri, Bishnupriya, and Mizo. The resources can be accessed here: Agentic Tool (https://github.com/PrachuryyaKaushik/AWED-FiNER), Web Application (https://hf.co/spaces/prachuryyaIITG/AWED-FiNER), and 53 Expert Detector Models (https://hf.co/collections/prachuryyaIITG/awed-finer).