Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Scalable Classification of Course Information Sheets Using Large Language Models".

Jane: Here's the gist of the paper. The gist The method presents an end-to-end pipeline for using large language models to scan course information sheets at scale, identify where assessments may be vulnerable to student use of GenAI tools, validate system performance through iterative refinement, and operationalise results through direct stakeholder communication and effort<ref:2603.13562#pg2> The research developed a four-phase pipeline for institutional risk assessment<ref:2603.13562#pg5>.

Tom: This pipeline included (0) manual pilot sampling, (1) iterative prompt engineering with multi-model comparison, (2) full production scan of 4,684 Bachelor and Master course information sheets from the Vrije Universiteit Brussel (VUB) with automated report generation and email distribution to teaching teams (91.4% address-matched) using a three-tier risk taxonomy (Clear risk, Potential risk, Low risk), and (3) longitudinal re-scan of 4,675 sheets after the next catalogue release<ref:2603.13562#pg10>.

Jane: The production scan involved processing HTML content extracted using Python with BeautifulSoup for parsing to extract raw HTML content; metadata (course ID, title, programme affiliation, ECTS credits, semester, language); and teaching team information<ref:2603.13562#pg8>.

Tom: Risk Classification and Model Selection A three-tier risk taxonomy was defined to categorize course sheets according to their exposure to risks associated with student GenAI use<ref:2603.13562#pg5>.

Title and authors: Tom: So, we're talking about this paper today, "Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance." It sounds like they built a way to use AI to check course sheets at a massive scale.

Jane: Right. It's about taking these huge piles of course information sheets and using large language models to quickly see where assessments might be vulnerable to students using GenAI tools, and then making that output useful for the teaching teams themselves.

Lu: Basically, they developed an end-to-end pipeline for this risk assessment. It starts with a manual pilot, then it moves into iterative prompt engineering with different models to get better results, and finally they do a full production scan of four thousand six hundred eighty-four sheets from Vrije Universiteit Brussel <ref:2603.13562#pg1>.

Meng: four thousand six hundred eighty-four sheets? That’s a lot of data for one project. What exactly are they scanning on each sheet? Is it just the course title, or does it pull in more detail?

Tom: It pulls in metadata like the course ID, the title, programme affiliation, ECTS credits, semester, and language information from HTML content extracted using Python with BeautifulSoup for parsing >

Jane: So they’re turning messy HTML into structured data that an AI can actually read and analyze. And they use this structure to apply a three-tier risk taxonomy: Clear risk, Potential risk, and Low risk >

Lu: The key thing is how they pick the model. They tested GPT-4o, o3-mini, and DeepSeek R1 in the pilot phase, but GPT-4o ended up being chosen for production because it was better at handling ambiguous cases and had more consistent output formatting > <ref:2603.13562#pg3>

Meng: Consistent formatting is important when you're shipping something to real teams. So they prioritized that consistency over just trying to beat a benchmark score >

Tom: Exactly. And then they did this iterative prompt engineering, cycling through five cycles based on manual labels, and they found two main issues: language inconsistency and being overly cautious in the AI's response >

Jane: It sounds like it was a process of learning how to talk to the AI over and over again until it started giving reliable results for what matters in this context >

Lu: They decided that for this method, they prioritized operational validity—meaning whether it actually works in an institution—over just getting the highest benchmark performance against gold standards >

Meng: That makes sense. If the output is good but takes three hours to clean up after, it doesn't help a busy department. So they focused on things that are immediately usable >

Tom: And then they moved into the production scan where they generated automated reports and emailed them to teaching teams, aiming for about ninety-one point four percent address-matched courses >

Title and authors: Jane: That’s a pretty high match rate for a massive catalogue. So, what were the results of that initial full scan? Where did most of the risk fall?

Lu: In Year one they found that sixty point three percent of courses were classified as Clear risk, fifteen point two percent as Potential risk, and twenty-four point five percent as Low risk across all those four thousand six hundred eighty-four courses > <ref:2603.13562#pg1>

Meng: Sixty point three percent is a significant amount of clear risk. That’s the immediate problem they were trying to solve, right? What does that mean for the actual teaching staff?

Tom: It means they have a lot of high-risk areas needing attention immediately, and then they did a longitudinal re-scan after the next catalogue release >

Jane: They tracked things over time. The comparison between Year one and Year two showed some important shifts in risk distributions across the courses > <ref:2603.13562#pg1>

Lu: They saw that improvements were most noticeable in programmes that are practice-oriented, like physical education, physiotherapy, and urban planning >

Meng: So the practical courses seem to be where the AI is actually helping them refine their understanding of what's risky. That’s a useful signal for deployment >

Tom: They noted that the overall Clear risk percentage actually went down from sixty point three percent in Year one to forty-eight point two percent in Year two > <ref:2603.13562#pg1>

Jane: That’s a substantial drop, especially in those practice-oriented programmes, so they think the refinement process is working on its own >

Lu: The paper also points out some specific challenges that came up during this process. They found two recurring case types that were hard to classify consistently: internship confusion and physical or practical requirements >

Meng: Internship confusion sounds like a tricky one. Are they saying that the location of the internship—whether it's off-campus or on-campus—is messing with the risk classification?

Tom: Yes, they found that initially, off-campus locations were treated as high risk, but this was refined >

Jane: And another issue was physical requirements, where some laboratory courses were misclassified because the written assignment components looked misleading >

Lu: The paper also makes a point that the AI classifies based on what’s written on the course sheet text, not what actually happens in practice >

Meng: So they're saying there could be a mismatch between documentation and reality, which is something production systems need to account for >

Tom: That’s the core limitation they flag—the LLM classifies based on documentation, not actual course practice >

Jane: So what does this mean for institutions that want to use this method? What are the next steps they suggest after seeing those results?

Lu: The paper suggests establishing a validation protocol with a diverse pilot sample, something with n greater than or equal to thirty courses, and planning for four to six iteration cycles with systematic error categorization >

Title and authors: Meng: So they’re telling institutions not just to run it once, but to build a rigorous testing loop around the entire thing >

Tom: And they emphasize that there needs to be uncertainty communication. They suggest that institutions must include explicit caveats in all automated communications and provide clear escalation paths for any classifications that are contested >

Jane: It sounds like they’re stressing the need for transparency when you deploy these kinds of systems, especially when the AI makes a judgment >

Lu: They also suggest maintaining version control on those prompts and using sensitivity analysis with matched subsets to bound pipeline confounds during longitudinal comparisons >

Meng: So they are saying don't just look at one number; track how the measurement process itself changes over time >

Tom: It really highlights that there’s always a tension between scale and accuracy here, between standardisation and contextual sensitivity, and between efficiency gains and the need for human judgment >

Jane: That seems to be the big picture they are trying to communicate about this paper. So how do we wrap up what we've seen?

Lu: This paper shows that you can take heterogeneous catalogue data and turn it into structured, actionable intelligence quickly, which is a huge step for quality assurance >

Meng: It’s a template, really. It shows how to move from massive amounts of raw text into something concrete that teaching teams can actually use to plan their course design >

Tom: The paper also suggests this method isn't just for GenAI risk; it’s a transferable approach for auditing other domains in higher education administration >

Jane: So, to wrap up, the work "Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance" gives institutions a reusable tool to identify risks at scale by using LLMs, but it also warns us about documentation mismatches and the need for human oversight in deployment.

Lu: The structure of the four phases—sampling, prompt engineering, production scan, and longitudinal comparison—is what makes it reusable rather than just a one-off experiment >

Meng: And for me as someone shipping models to production, the emphasis on operational validity over abstract benchmark perfection is exactly what you need when you’re building something that needs to run reliably every day >

Tom: It’s about making sure the AI output isn't just interesting in a lab, but actually helps someone make a decision about their course catalogue >

Jane: And that's what we have here. A method for using large language models to audit course sheets at scale, with clear steps on how to refine the prompts and what risks to watch out for when you deploy it into your institution.

The paper's summary: Tom: So we just finished looking at the mechanics of how they built this pipeline for course sheets, and now Jane you can give us the plain English summary of what all this actually achieves for an institution?

Jane: Right. So basically, they took a massive, messy database of course information—thousands and thousands of pages—and used a large language model to systematically flag which courses might be vulnerable to students using generative AI for assessments.

Tom: So it’s not just guessing; it’s a structured process. They defined these three risk levels: Clear risk, Potential risk, and Low risk. It makes the abstract idea of "AI use" into something they can actually categorize and manage on their end of the institution.

Jane: Exactly. The real win here is that it's designed to be reusable, not just a one-off experiment. They built a whole four-phase pipeline: you start with a small manual test, then you refine your instructions for the AI, run it on thousands of sheets at once, and then they even planned for checking the results again later when the course catalogue updates.

Tom: I like that iterative part. It shows they weren't just throwing a prompt at a black box expecting perfection. They actually identified specific problems, like inconsistent language or being too cautious in the AI's answers, and they fixed those through five cycles of refinement.

Jane: And that moves it from a science project to something operational. They showed that by prioritizing what works for real-world use—the operational validity—they got much better results than just chasing some abstract benchmark score.

Tom: That’s the key difference for me as someone who ships models, prioritizing fitness for use over theoretical perfection makes all the difference in production environments. What they found in their first year scan was pretty sobering, though.

Jane: They found that sixty point three percent of courses were flagged as Clear risk initially. That's a lot of high-risk areas to address right away in any curriculum audit, and then they showed that over the next year, when they re-scanned everything, that clear risk percentage dropped to forty-eight point two percent.

Tom: So the iterative refinement actually worked on its own? That’s interesting because it suggests the process itself is learning and improving how it reads those documents. But what about those tricky cases they found—the internship confusion and the physical requirements? Did they solve those?

Jane: Not completely, though. They admitted that some problems, like classifying an internship based on where the work is done off-campus versus on-campus, still confuse the system. And because the AI only reads what’s written on the sheet text—not what actually happens in practice—they flagged that as a major limitation of using this method alone.

Tom: That points to a huge caveat: you can't just trust documentation. If the course sheet is poorly written, even an advanced LLM will misclassify it because it’s basing its judgment on text, not reality. And that’s something we need to keep in mind when we build these systems for real use.

Jane: Absolutely. So what does this mean for someone who isn't an AI engineer? It means institutions can finally get a systematic way to move away from just guessing about GenAI risk in their courses, using a repeatable process that includes testing and refinement over time.

Tom: It gives them a template, then. A blueprint for how to take huge amounts of unstructured educational data and turn it into something structured and actionable for course design teams. And the authors even suggested things like needing explicit warnings in all automated reports so people don't just blindly trust the classification.

Jane: It’s a practical tool, really. It shows that while we have these powerful models, we still need a rigorous process around them to make sure they are actually safe and useful when you deploy them in a serious setting like higher education administration.

The paper's improvements: Tom: So we’re talking now about how they suggest you actually make this method better, since it’s not perfect right out of the box, and Jane can walk us through those fixes.

Jane: Right. The authors aren't just leaving you with a finished pipeline; they give you a set of concrete steps to improve the whole system, focusing on making it more robust for real use.

Tom: One big thing they suggest is establishing this formal validation protocol, which means you need a diverse pilot sample, at least thirty courses, and then plan for four to six iteration cycles where you systematically categorize any errors that pop up.

Jane: So it’s not just running the scan once; it’s building a loop where you actively track what breaks and how to fix those specific failure modes over several rounds of testing.

Tom: I like that idea of systematic error categorization. It means you aren't just looking at a random mistake; you are isolating the source of the problem so you can tune the prompt or change your data parsing specifically for that issue.

Jane: And they really emphasize communication, telling institutions they have to include explicit caveats in all automated emails and providing clear paths for when a classification is contested by a human. It’s about managing expectations upfront.

Tom: That makes sense from an engineering standpoint. You can't just let the AI make a final call without saying, "Hey, this is an educated guess based on the text." That uncertainty communication is crucial for deployment safety.

Jane: And they also stress version control for those prompts, so you know exactly which version of your instruction set led to a specific result later on. That keeps things traceable through time.

Tom: Yeah, and then there’s this part about using sensitivity analysis with matched subsets to bound pipeline confounds during those longitudinal comparisons. It means you have to be very careful when comparing Year one results to Year two results, because the method itself might be subtly changing how it measures things over time.

Jane: Exactly. They are reminding us that the measurement process itself can drift, so you need tools to check if the change in risk classification is actually due to a real change in what’s happening in the courses, or just because your AI setup got slightly different.

Tom: It sounds like they are really pushing for caution against making this a simple black box solution. They’re telling users that there's always tension between the efficiency of automation and the need for human judgment when you're dealing with complex things like academic risk assessment.

Jane: And that tension is what makes this paper important. It shows you how to use scale to get an overview, but it also forces you to acknowledge where the system stops working—like those internship or practical requirement confusion we talked about before.

Tom: So the implication for me is that if you want this method to actually be useful in a university setting, it can't just be about getting a high percentage; it has to be built with these iterative refinement and uncertainty handling steps baked into its DNA.

Jane: It changes things for someone listening because it shifts the focus from just "can we automate this?" to "how do we build this so that when it flags something, we know exactly how reliable that flag is?"

Conclusion: Tom: So we’re wrapping up this look at "Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance." Basically, they built a way to use AI to automatically flag course sheets for potential student GenAI misuse at a massive scale.

Jane: Right. The main thing is that it gives an institution a repeatable process—a pipeline—to take huge amounts of messy catalogue data and turn it into structured risk intelligence, which is pretty useful for quality assurance.

Lu: I think the real contribution here is showing that you can move beyond just one-off testing and build something modular. The fact that they focused on prompt engineering cycles and comparison across different models shows a deep understanding of how to tune the AI for a specific task.

Meng: From a deployment standpoint, it’s good because it addresses scale, but I still have to wonder what the actual operational cost is for running all that parsing and emailing at that volume.

Lalam: I’m pretty confident in the answer here; based on my literature review, this approach is sound because the methodology focuses on operational validity rather than just chasing a high score.

Tom: Operational validity, yeah. So they emphasize fitness for immediate use over abstract benchmark performance, which is exactly what you want when you’re trying to ship something into a real system.

Jane: Exactly. And they pointed out that the system needs uncertainty communication built in, telling people clearly when the AI is making a guess and providing escalation paths for those contested classifications.

Lu: That uncertainty handling is key because it acknowledges that the LLM classifies based on text documentation rather than actual course practice, which is a major limitation you have to be aware of.

Meng: Yeah, so the method stops working where the documentation is bad; it won't catch a good AI student if the sheet doesn't mention their specific assessment style. That’s a hard boundary for this approach.

Tom: It also highlights that there are inherent tensions in automating educational analysis between scale and accuracy, and between efficiency and the need for human judgment. That’s the reality of using any LLM here.

Jane: So, to summarize, "Scalable Classification of Course Information Sheets Using Large Language Models: A Reusable Institutional Method for Academic Quality Assurance" gives institutions a structured way to audit course sheets at scale while being very clear about where the AI’s limitations lie.

Lu: It’s a template for responsible deployment in higher education administration that moves beyond just the novelty of using LLMs and focuses on how to make that use reliable over time.

Meng: I just hope institutions actually follow through with those iteration cycles; a one-time run won't fix the underlying data quality issues.

Lalam: I’m sure they will, because the structure they put in place is designed for continuous improvement, not just a single report.

Tom: Well, that’s our wrap-up for this paper. We’ve seen how you can take messy catalogue data and turn it into actionable risk intelligence using iterative AI refinement.

Jane: Next up, we’re diving into something totally different—a paper on domain shift in echocardiography—so stick around.