Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts

The gist The authors present a domain-adapted Retrieval-Augmented Generation (RAG) pipeline for annotating pedagogical dialogue acts, achieving high Cohen’s κ scores by adapting the retrieval component rather than fine-tuning the generative model.

How it works

The proposed method employs a three-stage retrieval-augmented annotation pipeline that combines domain adapted embedding, utterance level indexing with parent chunk context, and codebook grounded in-context learning with a frozen LLM The task is to predict the label for each tutor utterance by constructing a retrieval index from the corpus, dynamically building context windows around target utterances, and passing both local context and retrieved examples to a frozen LLM for in-context classification All domain adaptation resides in the retriever that makes the pipeline portable across LLM backbones and reusable as new models become available

The pipeline involves several key steps, including:

1. Fine-tuning a lightweight sentence embedding model on tutoring dialogue corpora with Multiple Negatives Ranking Loss (MNRL), adapting the representation space so that utterances serving similar pedagogical functions cluster together regardless of surface lexical variation

2. Indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations

3. Presenting retrieved examples as few-shot demonstrations alongside the annotation codebook to a general purpose LLM, which performs classification through in context learning (ICL)

Key Components and Strategies

The methodology is built upon several specific technical choices designed to address the challenges of abstract and contextual tutoring moves

Semantic Chunking & Indexing:

The corpus is partitioned into semantically coherent chunks that preserve label homogeneity while respecting session boundaries Semantic boundaries are detected by computing the smoothed cosine similarity between overlapping context windows The boundary threshold τ is determined by leveraging sparse ground truth annotations and sweeping τ over [0.3, 1.0) to maximize F1 on this boundary classification problem

Domain-Adapted Embeddings:

The sentence transformer model is fine-tuned on labeled utterances from the TalkMoves and Eedi training sets combined using MNRL This fine-tuning improves semantic separation of tutoring dialogue labels in the embedding space, and it is trained on both classroom and dyadic chat data to capture pedagogical function across interaction formats

RAG Index Construction:

Two types of FAISS indexes are constructed: a chunk-level index using Cohere embeddings and two BGE-based indexes The utterance-level index embeds each labeled utterance individually, retrieving the parent chunk at query time

Experimental Results and Findings

The experiments were conducted across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b)

Performance Gains:

The best configuration achieved Cohen’s κ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no retrieval baselines (κ = 0.275-0.413 and 0.160-0.410)

Retrieval Precision:

The top-1 label match rate under RAG FINETUNED UTT reaches 62.0% on TalkMoves and 73.1% on Eedi, compared to 39.7% and 52.9% for RAG NO FINETUNE This confirms that retrieval precision drives annotation quality

Driver of Improvement:

An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7% to 62.0% on TalkMoves and 52.9% to 73.1% on Eedi under domain adapted retrieval

Confound Isolation:

The comparison between RAG NO FINETUNE and RAG FINETUNED UTT shows that the large jump to RAG FINETUNED UTT (+0.085 on TalkMoves, +0.129 on Eedi for GPT-5.2) comes predominantly from switching the index granularity from chunks to individual utterances

Conclusion

Utterance-level indexing with parent chunk retrieval consistently outperforms chunk-level indexing by preserving label specific signal while providing rich conversational context Domain adaptation on the retriever side accounts for the majority of this gain, and the improvement pattern holds across models of varying capability Utterance-level indexing with parent chunk retrieval consistently outperforms chunk-level indexing by preserving label specific signal while providing rich conversational context Future directions include extending to nonmathematical tutoring domains, integrating active learning for iterative index improvement, multi turn annotation that captures dialogue level pedagogical strategies, and alternative training objectives that enforce inter label separation without post hoc corrections

Ethics Statement

We analyze de identified tutoring dialogue datasets in accordance with our Institutional Review Board (IRB)-approved protocol [BLIND FOR REVIEW] We follow IRB approved procedures for data storage and access and use these data solely to understand and improve dialogue annotation workflows Automated annotation of tutoring moves is intended to augment rather than replace expert human judgment We caution against deploying automated labels as ground truth without such verification, particularly for high stakes decisions such as teacher evaluation or student assessment The confidence scores produced by our pipeline support a human in the loop workflow in which uncertain predictions are flagged for manual review Our system relies on commercial and open weight LLMs accessed via API, which introduces dependence on external services and associated cost, latency, and data handling considerations Finally the label taxonomy was developed for mathematics tutoring in English and may not transfer to other languages, subject areas, or cultural contexts without careful adaptation and validation

References

Rishabh Agarwal et al. Many-shot in-context learning Bakhtawar Ahtisham et al. Ai annotation orchestration: Evaluating llm verifiers to improve the quality of llm annotations, 2025 Aliki Anagnostopoulou et al. Human and llm-based assessment of teaching acts in expert-led explanatory dialogues, In Proceedings of the 6th Workshop on Computational Approaches to Discourse, pp. 166–181, 2025 Parishad BehnamGhader et al. Llm2vec: Large language models are secretly powerful text encoders, In Proceedings of the 1st Conference on Language Modeling, 2024 Sinchana Ramakanth Bhat et al. Rethinking chunk size for long document retrieval: A multi-dataset analysis, arXiv preprint arXiv:2505.21700, 2025 Huiyao Chen et al. Retrieval-style in-context learning for few-shot hierarchical text classification, 2024 Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations?, In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607–15631, Toronto, Canada, July 2023 Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.870 Qingxiu Dong et al.

1. textbfDomain-Adapted Retrieval for Annotation Quality Improvement (RAG Pipeline): The core system can be improved by implementing a three-stage retrieval-augmented annotation pipeline that combines domainadapted embedding, utterance-level indexing with parent chunk context, and codebook-grounded in-context learning with a frozen LLM. This system achieves Cohen’s κ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines (κ = 0.275-0.413 and 0.160-41).

2. textbfUtterance-Level Indexing Strategy: Precise Signal Preservation: The system can be improved by utilizing an utterance-level indexing strategy that preserves label-specific signal during search while returning parent chunks as few-shot demonstrations. This strategy is shown to drive the largest gains, with top-1 label match rates improving from 39.7% to 62.0% on TalkMoves and 52.9% to 73.1% on Eedi under domain-adapted retrieval.

3. textbfContextual Prompt Construction: Dynamic Context Window Generation: The system should dynamically construct a context window for each target utterance by expanding backward and forward from the target, stopping at positions where "sim(i) < τ or at session boundaries, up to a maximum of 10 utterances in each direction. This ensures the LLM receives full surrounding dialogue context" tailored to the specific pedagogical function of the target utterance.

4. textbfEmbedding Adaptation: Fine-Tuning for Domain Specificity: The system can be enhanced by fine-tuning a lightweight embedding model, specifically BGE-large-en-v1.5 on labeled utterances from the TalkMoves and Eedi training sets combined, using Multiple Negatives Ranking Loss (MNRL) to adapt the representation space so that utterances serving similar pedagogical functions cluster together regardless of surface lexical variation.

5. textbfConfidence Score Triage: Reliable Human-in-the-Loop Workflow: The system can be improved by leveraging confidence scores, as they are shown to shift the distribution rightward, allowing for a triage mechanism where RAG FINETUNED UTT substantially improves the reliability of high-confidence predictions across all models, with Sonnet 4.6 achieving 0.896 Cohen’s κ on its confident subset.

Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's κ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines (κ= 0.275 - 0.413 and 0.160 - 0.410). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7% to 62.0% on TalkMoves and 52.9% to 73.1% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.