Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments".

Jane: Here's the gist of the paper. The gist The workflow spans (1) constructing solution keys, (2) developing detailed rubric-style grading keys used to guide the LLM, and (3) a grading procedure that combines automated scanning and anonymisation, multi-pass LLM scoring, automated consistency checks, and mandatory human verification<ref:2603.13083#pg2> Providing timely and individualised feedback on handwritten student work is highly beneficial for learning but difficult to achieve at scale<ref:2603.13083#pg2>.

Tom: This challenge has become more pressing as generative AI undermines the reliability of take-home assessments, shifting emphasis toward supervised, inclass evaluation<ref:2603.13083#pg2>.

Jane: Short open-ended handwritten responses remain essential for evaluating generative reasoning and structured problem solving, despite substantial operational costs in grading time and feedback timeliness<ref:2603.13083#pg3>.

Tom: The paper presents a complete, scalable workflow that integrates LLMs into the grading and assessment cycle for short, supervised, pen-and-paper assessments<ref:2603.13083#pg3>.

Paper summary: Tom: So we've looked at how this paper "Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments" proposes a complete workflow to grade handwritten math tests using AI. The thesis is that you can create a scalable, end-to-end system where an LLM assists in grading these short, pen-and-paper assessments.

Jane: It claims this approach is useful because providing timely and individualised feedback on handwritten work is hard at scale, especially now with generative AI messing with take-home tests. So they propose a way to evaluate generative reasoning and structured problem solving through this LLM assistance.

Lu: The paper outlines three main stages for the workflow: first constructing solution keys, second developing detailed rubric-style grading keys to guide the LLM, and third a procedure that combines automated scanning and anonymisation with multi-pass LLM scoring and human verification.

Meng: What's important is that they tested this on low-stakes formative checkpoints in two math courses to see if it actually works under realistic classroom conditions. They used a standardized answer sheet for processing.

Tom: They say the tests were brief, frequent, and structurally uniform, which created a consistent stream of comparable handwritten responses suitable for evaluating this workflow.

Jane: And they found that performance across those six tests was rescaled to contribute up to one point on top of the final exam grade scored out of twenty <ref:2603.13083#pg1>. This shows it's designed to fit into an existing grading structure.

Lu: The paper points out that while multiple-choice formats are efficient, they often lack direct evidence of solution processes or diagnostic misconceptions, which is why open-ended handwritten responses remain essential for method evaluation.

Meng: That makes sense because MC items can conflate reasoning with test-taking strategies or just recognizing the right answer. So this workflow targets that gap in assessment methods.

Tom: The core of what they claim is that LLM assistance can reduce grading time by approximately twenty-three percent while achieving agreement comparable to, and in several cases tighter than, fully manual grading.

Jane: And they noted that when the digital condition was used, the agreement between annotators and the LLM was comparable to or even higher than the agreement between two human annotators doing it manually.

Lu: The authors found that deviations in grade calibration were more tightly concentrated near zero in the digital setting compared to manual grading, which they attribute to an anchoring effect where the LLM grade acts as a stabilizing reference.

Meng: So what this means for deployment is that LLMs can be supportive tools that reduce workload and stabilize decisions, but you still need a human check for those tricky outliers.

Tom: It's about showing that carefully embedding human-in-the-loop LLM grading can substantially cut the workload while maintaining fairness and accuracy in these specific scenarios.

Jane: And this whole process is designed to give students feedback quickly, which is the ultimate goal they are aiming for by making assessment more efficient.

Conclusion: Tom: So looking at "Human-in-Loop LLM Grading for Handwritten Mathematics Assessments" by Vanhoyweghen and his team, the authors are showing that this workflow is a practical way to use AI for grading handwritten work.

Jane: The implication is that we should view AI not as a replacement for human graders but as a supportive tool that can handle the volume of work in specific assessment settings while ensuring things remain accurate.

Lu: In simpler terms, it means you set up the keys and guide the AI with detailed rules, and then you use automated checks to catch mistakes before a person has to look at every single answer.

Meng: For me, it suggests that if we want to deploy this kind of thing in production, we need that human verification step built right into the pipeline for those moments when the AI is uncertain.

Tom: The authors are showing how you can get significant efficiency gains without sacrificing fairness, provided you keep that human-in-the-loop element active for those rare outlier cases.

Jane: This paper suggests a shift in how we think about assessment: instead of just thinking about what the AI *can* grade, we focus on building systems that make grading faster and more consistent for students.

Lu: The workflow demonstrates that this setup is not specific to math; it could be extended to other subjects where short, structured tasks require showing incremental understanding through steps.

Meng: I'm interested in the future work here—basically figuring out how to make this pipeline robust enough for real-world, messy student submissions beyond the controlled test sheets they used.

Tom: That sounds like the next logical step: moving from a controlled environment to a more general application where the system can handle anything.