Beyond Benchmarking: Scenario-Based Evaluation of Large Language Models for Personalized Learning

The study presents an empirical comparison of three state-of-the-art LLMs on a tutoring task simulating a realistic learning setting. This research addresses the scarcity of systematic head-to-head evaluations in authentic learning scenarios despite the growing use of Large Language Models (LLMs) as potential tools for personalized education.

Methodology and Experimental Design:

To simulate this realistic scenario, the researchers utilized a dataset containing a student’s responses to ten mixed-format questions with correctness labels. The LLMs were tasked with three specific functions: (i) analyze the quiz to identify underlying knowledge components, (ii) infer the student’s mastery profile, and (iii) generate targeted guidance for improvement.

To ensure rigor and mitigate subjectivity, Gemini was employed as a virtual judge to perform pairwise comparisons across multiple dimensions: accuracy, clarity, actionability, and appropriateness. The evaluation relied on a strict protocol where Gemini determined the superior model based on five pedagogical criteria: accuracy of knowledge diagnosis, specificity and actionability of feedback, identification of misconceptions, instructional clarity, and appropriateness to the student’s current level.

Quantitative Analysis:

The results were analyzed using the Bradley–Terry (BT) model, which is a probabilistic framework used for paired comparison data. The study conducted 30 pairwise evaluations across 10 runs.

The quantitative analysis revealed a clear performance hierarchy: "GPT-4o > DeepSeek-V3 > GLM-4.5. Specifically, the BT strength estimates showed that GPT-4o achieved the highest strength (BT = 0.95), indicating statistically significant superiority, while GLM-4.5 was consistently weaker" (BT = -1.32).

Findings and Conclusion:

The study found that GPT-4o is generally preferred, producing feedback that is more informative and better structured than its counterparts, whereas DeepSeek-V3 and GLM-4.5 demonstrate intermittent strengths but lower consistency.

The findings collectively demonstrate the feasibility of deploying LLMs as advanced teaching assistants for individualized support. Furthermore, the research provides methodological insights for subsequent empirical research on LLM-driven personalized learning, emphasizing that future evaluation must move beyond simple correctness and incorporate factors such as clarity, diagnostic depth, actionability, and communicative tone to capture their true educational value.

The following improvements are derived directly from synthesizing the methodological gaps identified in the provided literature review. Given the high-stakes nature of AI deployment in education, these upgrades focus not just on capability, but critically on verifiable pedagogical quality and adaptability.

The system must transition from merely assessing correctness to providing comprehensive, structured feedback that models human pedagogical expertise. This requires building a dedicated, modular Feedback Generation Layer separate from the core knowledge retrieval mechanism.

• Specific Improvement: Integration of four distinct, weighted feedback vectors:

1. Diagnostic Depth Score: Identifies the root cause of an error (e.g., conceptual misunderstanding vs. procedural oversight).

2. Actionability Metric: Generates concrete, step-by-step remediation tasks linked directly to required prerequisite knowledge gaps (e.g., Review Section 2.1 on basic calculus).

3. Clarity & Tone Scorer: Adjusts the linguistic register and tone of the feedback based on the user profile (see Point 4), ensuring it is encouraging, non-judgmental, and academically precise.

4. Communicative Tone Index: Allows explicit tuning of formality (e.g., Socratic questioning vs. direct instruction) to match instructor preference or student maturity level.

The system must be architected to handle extreme input variance, moving beyond single-domain expertise. This requires an abstract representation layer that separates content knowledge from pedagogical scaffolding.

• Specific Improvement: Implementation of a Cross-Lingual and Cross-Domain Adaptation Module (CLCDAM).

• Mechanism: The system will maintain separate, parameterized models for cultural norms and pedagogical conventions. When deployed in a new context (e.g., different country, different academic discipline), the CLCDAM acts as a pre-processor, fine-tuning the output structure to align with local educational standards (e.g., shifting emphasis from individual achievement to collaborative learning outcomes, or adjusting formality levels based on regional norms).

• Input Handling: The system must accept and process mixed modalities of input (e.g., code snippets alongside written explanations, or diagrams alongside textual questions) and treat them as equally weighted components of the student's performance profile.

The AI cannot operate in a vacuum; it must iteratively learn from expert human judgment to refine its own pedagogical weights.

• Specific Improvement: A mandatory, structured feedback loop where the AI submits its proposed evaluation/feedback to an instructor/student panel for scoring and qualitative review.

• Mechanism: The system logs discrepancies between its output and the human consensus (e.g., AI suggested remediation X, but Instructor rated Y as superior). These discrepancies are then used in a continuous retraining cycle to refine the internal weighting of effectiveness or appropriateness. This directly addresses the need for instructor calibration observed in advanced research.

The system must move beyond simple difficulty tracking to model deep, multi-faceted learner profiles that dictate the style of intervention.

• Specific Improvement: Implementation of a Dynamic Cognitive Style Profiler (DCSP).

• Mechanism: The DCSP continuously analyzes student interaction patterns—not just right/wrong answers, but time spent, number of attempts, preferred question format, and self-correction behavior. This profile then triggers adaptive scaffolding:

• If the student profile indicates a Visual Learner, the feedback will prioritize flowcharts and conceptual mapping.

• If the profile indicates an Anxious/Low-Confidence Learner, the system will automatically lower its diagnostic depth score initially, focusing only on positive reinforcement and small, achievable micro-wins to build self-efficacy before tackling complex errors.

• This allows the system to determine if a feedback style is universally effective or requires precise adaptation to the individual's cognitive profile.

The resulting system will not merely grade assignments; it will function as a Cognitively Adaptive, Culturally Aware, and Iteratively Calibrated Digital Tutor.

1. Provide Hyper-Personalized Remediation: It can diagnose why a student failed (e.g., You misunderstood the relationship between X and Y) rather than just stating that the answer was incorrect. It will then generate targeted micro-lessons using the student's preferred modality (visual, textual, or interactive coding example) until mastery is achieved.

2. Guarantee Pedagogical Robustness: Its feedback is guaranteed to be comprehensive, covering clarity, actionability, and tone—a trifecta often missing in current academic AI tools.

3. Operate Globally and Contextually: It can seamlessly transition between educational systems and languages while adjusting its underlying assumptions about what constitutes good learning practice for that specific context.

4. Self-Improve Through Expertise: By integrating the HITL framework, the system perpetually improves its pedagogical model by incorporating real-world expert judgment, ensuring it remains methodologically sound and highly trustworthy for high-stakes educational environments.