HybridRAG-BN: A Retrieval-Augmented Framework with Fine-Tuned Verification for Bangla KBQA

HybridRAG-BN is a retrieval-augmented framework for Bangla knowledge-base question answering (KBQA) that integrates hybrid retrieval using BM25 and BGE-M3, answer generation using the GGUF version of Gemma-4-31B-Instruct, and a LoRA-fine-tuned Gemma-4-31B-Instruct model for answer verification and refinement. To further improve robustness, the framework incorporates a post-processing stage that addresses unresolved cases through fallback answer replacement and DuckDuckGo-assisted retrieval. Experimental results demonstrate the effectiveness of the proposed framework, achieving token-level F1 scores of 0.71654 and 0.72912 on the public and private leaderboards, respectively, securing first place in the competition.

The framework consists of four stages: hybrid retrieval using BM25 and BGE-M3 to combine lexical and semantic matching; answer generation using the GGUF version of Gemma-4-31B-Instruct; answer verification and refinement using a LoRA fine-tuned Gemma-4-31B-Instruct model; and post-processing through fallback answer replacement and external DuckDuckGo-assisted retrieval for unresolved cases.

Two knowledge-base preprocessing and chunking strategies were investigated. Approach 1 – Precision-Based: The raw Bangla Wikipedia knowledge base was reconstructed at the page level, cleaned by removing recurring boilerplate sections (headers, footers, navigation blocks, and language-list sections), and divided into overlapping text chunks with a target length of approximately 1000 characters and an overlap of 150 characters. Approach 2 – Coverage-Based: A less aggressive preprocessing pipeline that removed only the header block, footer block, and the first type of tool/navigation block, with a reduced target chunk size of approximately 800 characters while maintaining an overlap of 150 characters.

For retrieval, all knowledge-base chunks were encoded using the BGE-M3 embedding model and indexed using FAISS. A hybrid retrieval strategy employed BM25 for keyword overlap and dense retrieval with BGE-M3, combining candidate passages through weighted score fusion (BM25 weight 0.65, dense weight 0.35). The candidate passages were further refined using the BGE-Reranker-v2-M3 cross-encoder model, and the passage immediately preceding each selected chunk was included to preserve local contextual information.

For inference, Approach 1 used the aggressively cleaned knowledge base with a prompt that required the model to return Context-এ তথয্ েনই when sufficient evidence could not be identified, enabling unresolved cases to be identified during post-processing. Approach 2 used the less aggressive preprocessing strategy with a different prompt that discouraged abstention-style responses and instructed the model to use its pretrained knowledge to generate the most plausible answer when the retrieved context was insufficient. A two-stage retrieval strategy was employed, with retry triggered when the generated answer indicated insufficient contextual evidence, using an expanded candidate pool (k = 10).

The proposed fine-tuning framework uses Approach 2 as its primary pipeline, followed by fine-tuned answer verification, Approach 1-based fallback replacement, and DuckDuckGo-assisted retrieval for remaining unresolved cases. The fine-tuned model was trained using supervised fine-tuning with TRL's SFTTrainer and LoRA adaptation on a 4-bit quantized Gemma-4-31B-Instruct model, with response-only training. The training dataset consisted of questions, ground-truth answers, and reasoning annotations, divided into training and validation subsets using an 80/20 split. The fine-tuning configuration included LoRA rank 16, LoRA alpha 32, learning rate 1.5 × 10−5, 2 epochs, and 122.4M trainable parameters (0.39%).

During judge-based inference, the fine-tuned model receives the original question, retrieved context, and candidate answer produced by Approach 2, and evaluates whether the answer is correct and supported by the available evidence. The maximum generation length was limited to 30 tokens to encourage concise refinements. Post-processing involved rule-based filtering to identify abstention-style responses, replacing them with Approach 1 predictions when available, and using DuckDuckGo-assisted retrieval with entity extraction for remaining unresolved cases.

Model scaling experiments evaluated three other Gemma 4 instruction-tuned models: Gemma-4-E2B-it, Gemma-4-E4B-it, and Gemma-4-26B-A4B-it. Results showed a clear positive relationship between model scale and question-answering performance, with public F1 scores of 0.59634, 0.60247, and 0.66393 respectively, and private F1 scores of 0.60925, 0.60727, and 0.68064 respectively. Based on this trend, Gemma-4-31B-Instruct was selected as the foundation of the proposed framework.

The overall performance comparison showed that Approach 1 achieved public F1 of 0.69329 and private F1 of 0.69147, Approach 2 achieved public F1 of 0.70223 and private F1 of 0.69901, Approach 2 + Fine-Tuned Verification achieved public F1 of 0.71589 and private F1 of 0.72495, and the proposed framework achieved public F1 of 0.71654 and private F1 of 0.72912. The less aggressive preprocessing strategy improved retrieval effectiveness, while the fine-tuned verification and post-processing stages further improved answer quality.

The fine-tuned verification model modified 48 of the 1,500 generated answers, including factual corrections, improved entity resolution, and more precise answer extraction, producing a substantial improvement in token-level F1. The post-processing stage affected 11 predictions with abstention-style responses, with four being correct answers from Approach 1 replacement. Seven predictions still contained the response Context-এ তথয্ েনই after replacement, and five were successfully converted into answer-bearing predictions through DuckDuckGo-assisted retrieval.

The framework has several limitations: the fine-tuned verification model was constrained to a maximum generation length of 30 tokens, occasionally resulting in truncated answers for list-type questions; the framework relies heavily on Gemma-4-31B-Instruct across multiple pipeline stages, making it computationally expensive; and model scaling experiments were conducted exclusively within the Gemma 4 family, leaving generalizability to other model families such as Llama, Qwen, or Mistral unexplored.

Improvements to AI Systems Based on This Paper:

1. Hybrid Retrieval with Weighted Fusion and Context Padding: Implement a retrieval system that combines lexical (BM25) and dense (BGE-M3) retrieval with a weighted score fusion (0.65/0.35), followed by a cross-encoder reranker (BGE-Reranker-v2-M3). Additionally, include the passage immediately preceding each selected chunk to preserve local context. This improves answer accuracy for low-resource languages by balancing keyword precision with semantic understanding.

2. Adaptive Abstention and Retry Mechanism: Build an AI system that can explicitly signal insufficient context (e.g., Context-এ তথ্য নেই) and trigger a two-stage retrieval retry with an expanded candidate pool (k=10) when such abstention is detected. This reduces hallucination and improves robustness for out-of-knowledge-base queries.

3. Fine-Tuned Verification and Refinement Layer: Add a LoRA-fine-tuned LLM (e.g., Gemma-4-31B-Instruct) as a separate verification stage that receives the original question, retrieved context, and candidate answer, then outputs a concise correction (max 30 tokens). This layer improves factual accuracy, entity resolution, and answer extraction, as demonstrated by a 0.013–0.026 F1 gain.

4. Post-Processing with Fallback and External Retrieval: Implement a rule-based post-processing stage that (a) detects abstention-style responses, (b) replaces them with predictions from a more aggressive preprocessing pipeline (Approach 1), and (c) uses DuckDuckGo-assisted retrieval with entity extraction for remaining unresolved cases. This converts abstentions into answer-bearing predictions, improving overall coverage.

5. Preprocessing Strategy Selection Based on Retrieval Goal: Use a less aggressive preprocessing pipeline (removing only headers, footers, and first navigation block) with smaller chunk sizes (800 chars) for primary retrieval, as it outperforms aggressive cleaning. Reserve aggressive cleaning (page-level reconstruction, removing all boilerplate) for fallback scenarios. This optimizes both recall and precision.

6. Scaled Model Selection via Empirical Trend: When choosing a foundation model, run scaling experiments across multiple sizes within a family (e.g., Gemma-4-E2B, E4B, 26B-A4B, 31B) to identify the performance-to-compute sweet spot. Use the observed positive scaling trend to justify selecting the largest feasible model, improving answer quality.

What the Improved AI System Can Do:

• Answer Bangla knowledge-base questions with higher token-level F1 (0.72912 private) by combining lexical and semantic retrieval, reranking, and context padding.

• Avoid hallucination by explicitly abstaining when evidence is insufficient, then retrying with broader retrieval or falling back to external web search.

• Self-correct answers through a fine-tuned verification layer that refines entity names, fixes factual errors, and extracts more precise answers.

• Handle diverse query types (list-type, multi-hop) with adaptive chunk sizes and context windows, though truncation remains a limitation for long lists.

• Operate effectively in low-resource languages without requiring massive labeled datasets, using LoRA fine-tuning on a quantized base model (0.39% trainable parameters).

• Provide a modular pipeline where each stage (retrieval, generation, verification, post-processing) can be independently improved or swapped, enabling rapid adaptation to new domains or languages.