Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos

This paper details an evaluation of a course-grounded chatbot designed for STEM lecture videos, emphasizing strict adherence to provided source material. The methodology involves rigorous offline retrieval benchmarking and diagnostic audits to assess the system's ability to answer questions using only cited evidence. This work is crucial because it establishes quantitative proof of concept for grounding large language models in specific educational content, addressing the critical need for factual accuracy in academic AI tools.

Diagnostic Audit Findings and Failure Analysis

The evaluation included a separate diagnostic audit of 50 uncited messages. The results indicated that 54% of the queries clearly unanswerable from the available course evidence and 46% unclear. Regarding the chatbot's refusals, auditors rated 58% of the refusals appropriate, 38% unclear, and 4% inappropriate. The analysis identified consistent failure patterns: The dominant labeled failure reasons were insufficient evidence to judge and missing course content, at 40% each. A key limitation noted across both audits is that the judge’s inventory was assembled only from passages cited elsewhere in the same course; consequently, material that retrieval never surfaced was invisible to the judge, preventing an estimate of answerable-but-missed populations.

Offline Retrieval Benchmarking Setup

Because production courses lack gold retrieval labels, an offline benchmark utilizing EduVidQA was necessary. This evaluation corpus comprises 10 courses, 139 videos, and 5,924 chunks. Four distinct retrieval arms were compared: dense-only cosine retrieval; dense plus BM25 without the summary prior; the complete system adding the chaptersummary soft prior; and a control arm scoring all chunks without course isolation. All arms maintained standardized parameters, including 250-word chunks with 50-word overlap, using all-MiniLM-L6-v2 embeddings, a seven-chunk retrieval budget, and a 12,000-character context budget.

Performance Comparison of Retrieval Arms

The comparison between the dense-only retrieval and the complete system demonstrated significant performance gains across multiple metrics when tested over 269 real-world questions. Specifically:

• Timestamp hit improved from 0.4015 to 0.4572.

• Video hit improved from 0.6840 to 0.7472.

• MRR increased from 0.2504 to 0.2934.

• nDCG@5 increased from 0.2826 to 0.3299.

These improvements were statistically significant, with the complete system showing strong advantages in all measured metrics compared to the baseline retrieval method. Furthermore, removing course isolation was found to be detrimental, with significance noted for both video hit (p 0.0005) and timestamp hit (p 0.034).

Sensitivity and Stability Analysis

The system's performance was tested for sensitivity by varying the prior weight (gamma) while keeping the dense-to-BM25 ratio constant. The results indicated that Performance was strongest at gamma = 0.1 and generally declined as the chapter-summary prior received more weight. This suggests that the prior acts as a modest refinement rather than the load-bearing component. On the synthetic split, the complete system improved video hit from 0.834 to 0.860. The evaluation was also rigorously tested for reproducibility, running two separate processes across 807 query-arm comparisons; these runs produced zero mismatches in retrieved chunk identities, scores, per-query metrics, aggregate metrics, and significance inputs within a tolerance of 10-12.

1. Implement Multi-Stage, Weighted Prior Mechanisms (Inference Time Optimization)

• Improvement: Do not rely on a single retrieval method. Architect the system to use a weighted ensemble of retrieval priors:

• P dense: Standard dense vector cosine similarity (MiniLM-L6-v2).

• P keyword: BM25/Sparse matching (for term frequency and exact keyword hits).

• P contextual: A dedicated prior source (e.g., chapter summary, or metadata summaries) that is explicitly weighted (gamma) and combined with the dense/sparse results.

• Mechanism: The final retrieval score for a chunk C should be calculated as Score(C) = alpha times Score dense(C) + (1-alpha-beta) times (beta times Score BM25(C) + (gamma - 1) times Score contextual(C)). Crucially, alpha, beta, and gamma must be optimized per query type or domain, rather than fixed globally.

• System Capability: The system can dynamically select the optimal retrieval path (e.g., prioritizing P contextual for high-level concept questions, and P dense for specific factual recall) leading to significantly higher MRR and nDCG@5 scores across diverse query types, surpassing single-source baselines.

2. Incorporate Granular, Timestamp-Aware Retrieval (Precision & Recall)

• Improvement: The retrieval module must be trained and evaluated not just on chunk relevance (binary), but on temporal coverage. Each retrieved chunk should output a confidence score tied to its source video's timestamp range.

• Mechanism: Implement a specialized loss function during fine-tuning that penalizes missed gold timestamps (Loss TS) based on the distance and duration of the gap between the start of the query and the start/end times of retrieved chunks. The system must prioritize retrieving chunks that cover a continuous time segment around the expected answer point (the gold timestamp).

• System Capability: The AI can pinpoint answers with high temporal accuracy, directly citing not just what was said, but when it was said (e.g., According to the video at 12:34–12:50...), drastically improving verifiability and user trust.

3. Develop a Structured Missed Evidence Diagnostic Module (Self-Correction/Refusal)

• Improvement: Augment the refusal mechanism beyond simple confidence scoring. The system must maintain an internal Course Content Inventory (the judge's inventory) that tracks material available to the model but not retrieved.

• Mechanism: When retrieval fails, the module should perform a secondary diagnostic query against its known, but unsourced, curriculum knowledge base. If the failure is due to missing course content (a major labeled failure reason), instead of issuing a generic refusal, it should signal a Curriculum Gap Warning.

• System Capability: The AI system can differentiate between three types of failures: 1) Insufficient Evidence (the query is unanswerable given the source); 2) Retrieval Failure (the evidence exists but was missed); and 3) Curriculum Gap (the evidence exists but was not included in the initial corpus). This provides actionable feedback to the user/developer.

4. Implement Robust Stability and Reproducibility Controls (Engineering Best Practice)

• Improvement: Mandate strict, deterministic operational protocols for all scoring and retrieval passes.

• Mechanism: The entire pipeline must be run with a fixed random seed, pinned hardware configurations (e.g., CPU/GPU memory allocation), and defined tie-breaking rules based on immutable chunk identifiers. All core metric calculations (McNemar test, Wilcoxon signed-rank test) must be implemented as self-contained, verifiable functions that accept the full set of paired inputs.

• System Capability: The system guarantees perfect reproducibility of all performance metrics across separate processes and hardware environments, fulfilling the highest standards required for scientific publication and high-stakes deployment.

5. Enhance Behavioral Analysis for Course Design (Curriculum Optimization)

• Improvement: Integrate the findings from course-level behavioral contrasts (Table 11) into the curriculum design phase itself.

• Mechanism: The system should analyze input logs to correlate low Cited percentages with high Refusal Rate or high Homework-Paste Rate. If a specific course module has a low citation rate but a high failure rate, it suggests the material is either poorly understood or the source evidence is too sparse/difficult to retrieve.

• System Capability: The AI can guide instructional designers to revise course structure—for example, flagging that Course X needs more explicit textual markers or foundational review sections because students are failing questions where the provided evidence is scattered (low citation density).

Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask course specific questions or verify answers against an instructor's lecture. We report a semester-long deployment of VideoPoints platform with a retrieval-augmented chatbot that answers from course lecture materials and returns timestamped citations. The chatbot retrieves only from the active course, uses chapter summaries to guide transcript ranking, and returns clickable timestamped citations. Students used it for quick lookups and exam review. Across 833 messages, 70.5% included citations, none crossed a course boundary, and when no lecture evidence matched, the chatbot usually declined rather than answering. Among the users, citations were the most consistently useful feature, while practice-question generation was the strongest unmet request. We also evaluated the design on the real-world test split of EduVidQA, a public multimodal benchmark for lecture-video question answering. Our design improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval. Together, the results show that effective deployment depends on course isolation, supported citations, and alignment with students' study practices.