Gen AI in Proof-based Math Courses: A Pilot Study

arXiv:2509.13570 · cs.AI, math.HO · Submitted 2026-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Gen AI in Proof-based Math Courses: A Pilot Study".

Jane: The paper was written by the authors from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary and Implications: Jane: So, building on our discussion of the title, the paper summarizes a few key areas of their findings regarding how these AI tools actually interact with students.

Tom: They're moving past the initial idea that AI will just give away the answers, which is a relief to hear because that’s always the fear in EdTech.

Lu: The summary points toward using AI to generate specific types of feedback—feedback on *where* a student went wrong logically, rather than just telling them they are wrong.

Meng: I found the discussion about iterative refinement particularly interesting; it suggests that the AI is designed to handle multiple drafts and gradually guide the student closer to correctness.

Lalam: It highlights a shift in educational design, moving from assessment models—where you grade after the fact—to continuous, real-time co-creation with AI support.

Jane: The authors seem very careful not to overpromise on what this technology can achieve, which gives the findings a lot of credibility.

Tom: That’s right; they are presenting a nuanced view that acknowledges both the immense potential and the current limitations of these models in complex reasoning tasks.

Lu: They really emphasize that AI needs to understand not just mathematical syntax, but also pedagogical intent—the goal is learning, not just completion.

Meng: From an implementation standpoint, this suggests the system needs robust guardrails and a very granular understanding of the curriculum structure to be useful in a real classroom.

Lalam: And what that means for culture is that it validates a new model of knowledge transfer—one where the AI acts as a cognitive partner, elevating human potential rather than diminishing it.

Jane: It sounds like they are suggesting that the best use of this technology is to focus on metacognition—teaching students how to think about their own thinking.

Tom: This really changes the conversation from "Can AI solve math problems?" to "How can AI help us *learn* how to solve them?" That's a big pivot.

Lu: Now that we understand the scope of the findings, I’m incredibly curious about what improvements they are proposing for future implementations.

Suggested Improvements: Tom: Okay, so we’ve seen the initial results and understood the general summary; now we're looking at how the authors recommend improving this system in practice.

Jane: The suggestions seem to focus heavily on making the interaction feel more like a human mentor's guidance rather than just a prompt-response chat.

Meng: One major point they raise is integrating AI feedback directly into existing Learning Management Systems (LMS), making it seamless for the educator and student alike.

Lu: I think they also stress the importance of tailoring the difficulty level dynamically, which is critical because one size of AI scaffolding does not fit all learners.

Lalam: The most impactful suggestion, in my view, relates to how AI can help standardize the *experience* of learning across diverse educational settings globally.

Jane: So it’s not just about building a better model; it's about building a better *workflow* around that model for teachers and students to use.

Tom: It sounds like they are pushing for a shift towards hybrid models, where the AI enhances the teacher's efforts rather than replacing them entirely.

Meng: Practically speaking, making it adaptable across different mathematical domains—algebra versus topology—is going to require significant engineering effort and standardization of inputs.

Lu: And I think we need to consider how those improvements could also incorporate emotional intelligence, recognizing when a student is frustrated versus simply confused by the material.

Lalam: If we follow these suggested improvements, it fundamentally improves the cultural value placed on education—it makes learning feel more personalized and achievable for everyone.

Jane: So, in essence, they are arguing that the future of this technology lies in its ability to be context-aware and highly customizable based on individual student needs.

Tom: That gives us a fantastic roadmap for how this research can evolve from a pilot study into a widespread educational tool. But before we wrap up, let's synthesize what this all means for the big picture.

Conclusion: Jane: Wow, we’ve covered so much ground today discussing "Gen AI in Proof-based Math Courses: A Pilot Study." If I were to summarize the overall implication for our listeners, it's that AI is becoming a powerful assistant for complex reasoning.

Tom: And it’s not just about *getting* the answer; it’s about making the journey toward understanding—the struggle and refinement—visible and supportable by technology.

Lu: What this paper confirms is that generative AI has the potential to democratize access to high-level, personalized academic guidance, which is a monumental shift for global education.

Meng: From a practical standpoint, the success of these improvements means we need to start thinking about building enterprise-grade systems that are not only smart but also incredibly reliable and scalable in real institutions.

Lalam: The impact here extends beyond math; it models how AI can fundamentally improve human culture by making challenging, formerly gatekept knowledge accessible through personalized guidance.

Jane: It seems like the technology is ready to shift from "proof of concept" to genuine educational utility, provided the developers listen closely to these kinds of pilot studies.

Tom: So, we've seen that AI excels at scaffolding the process, helping students

Conclusion: Tom: So, after hearing all of this fascinating breakdown of "Gen AI in Proof-based Math Courses: A Pilot Study," it really makes you think about how fast things are changing in education.

Jane: It’s incredible, Tom. What struck me most is how the paper showed that these tools aren't just giving answers; they're actually helping students understand *why* the answer is correct.

Lu: Exactly, Jane! The generative nature means it can adapt its explanations based on where a student got stuck, which is a huge breakthrough for personalized learning paths.

Meng: But I gotta ask, Lu, even if the adaptive explanation is perfect, how do you actually build that system to handle the sheer complexity of mathematical proofs?

Jane: Well, I think what we're seeing here is a shift from simply consuming knowledge to actively collaborating with an intelligent tutor.

Tom: And that collaborative element is huge! It suggests AI might not replace the professor, but it’s definitely going to change the classroom dynamic entirely.

Lalam: The most profound implication, if I may chime in, isn't about math at all; it’s about democratizing high-level reasoning skills globally.

Lu: You hit on something important there, Lalam—it levels the playing field dramatically for people who don't have access to top-tier tutoring before.

Meng: From an engineering standpoint, the next step has to be rigorous validation across diverse curricula, because math changes a lot depending on the degree.

Jane: That's right, Meng; we need tools that are reliable and trustworthy when dealing with foundational concepts like proof structure.

Tom: It sounds like this isn't a single solution, but rather an entire ecosystem of AI support tools for academia.

Lalam: It fundamentally changes how culture views the process of learning—it validates struggle as part of the journey, not just getting the final answer right.

Lu: I can only imagine what other academic disciplines will follow this lead once they see how powerful this kind of scaffolding is.

Meng: Hopefully, we'll see some open-source models emerge soon so that smaller institutions can actually afford to implement this technology.

Jane: It's such an exciting time for pedagogy, isn't it? We really appreciate the insights from everyone today about "Gen AI in Proof-based Math Courses: A Pilot Study."

Tom: Absolutely! Thanks to all of you for breaking this down with us; we can't wait to hear what groundbreaking research is coming up next.

cs.AI, math.HO

Submitted: 2026-08-20

Updated: 2026-08-21

Importance score: 76/100

The gist: * Introduction and Context This study was motivated by the rapid rise of generative artificial intelligence (AI) in higher education, coupled with the unreliability of current AI detection tools.

Key concepts

Metacognition
This educational focus involves teaching students to be aware of, and think about, their own learning processes. The goal is not just getting the correct answer, but understanding *how* they arrived at that conclusion and recognizing where their thinking went wrong.
Real-time Co-creation
This describes a shift in educational design where learning is supported continuously by AI. Instead of grading work after it is finished, the AI assists students in the process, guiding them through multiple drafts and improving their understanding as they go.
Cognitive Partner
This suggests that AI should function as an assistant that elevates human intellectual potential rather than replacing the teacher or student. It helps students with complex reasoning tasks by scaffolding the process and supporting the journey toward understanding.

Terminology

Summary

Introduction and Context

This study was motivated by the rapid rise of generative artificial intelligence (AI) in higher education, coupled with the unreliability of current AI detection tools. The authors note that With the rapid rise of generative AI in higher education and the unreliability of current AI detection tools, developing policies that encourage student learning and critical thinking has become increasingly important. The research was conducted following a policy implementation at Indiana University East, which allowed for generative AI use but required students to be highly careful with referencing.

Study Scope and Methodology

The study examined student use and perceptions of generative AI across three specific proof-based undergraduate mathematics courses: a first-semester abstract algebra course, an undergraduate topology course, and a second-semester abstract algebra course. The data collection involved survey responses and student interviews.

The methodology included:

  1. Survey Data: A total of 19 survey responses were collected from 17 students across the three courses (M403, M404, and M421).

  2. Interview Data: Four students participated in follow-up interviews conducted via Zoom, which lasted approximately 15–20 minutes each and followed a semi-structured protocol.

The implemented Gen AI Policy required specific actions from the students:

  • For homework, students had to evaluate the response and provide appropriate justifications if the resulting solution or proof is correct and has sufficient work shown while only utilizing results learned in the course.

  • For discussion assignments, students were required to state which parts are generated by GAI unless you have only used it for grammar and spelling revisions.

Key Findings: Student Use and Perceptions

The findings reveal several nuanced patterns regarding student engagement with generative AI:

  • ** Usage:** Out of the 19 participants, 8 out of 19 participants reported they did not use generative AI for their coursework although using it was allowed by course policy.

  • Perception of Cheating: Significantly, 7 of the participants viewed utilizing generative AI for coursework as cheating.

  • How AI Was Utilized: Students who used the tool primarily utilized it for brainstorming, providing feedback, explaining concepts and references for further learning, rather than using it to solve problems directly. One student noted that it helped them go deeper into some of the history and the concepts.

  • Helpfulness: The survey data indicated that students generally found M.S. Copilot to be a helpful tool for learning in a proof-based mathematics course. However, the impact on independent problem-solving was more mixed, with responses distributed across 'yes,' 'no,' and 'not sure.'

  • Reliability: Students demonstrated a cautious and discerning approach to AI-generated content. Many were consistently vigilant in verifying the information it provided, often identifying inaccuracies.

Key Findings: Engagement and Practical Implications

The study observed the following regarding student interaction:

  • Instructor Interaction: The use of M.S. Copilot did not generally feel that using Copilot reduced their engagement with instructors.

  • Peer Interaction: The effects on peer collaboration were varied. While some students felt that AI use elevated the quality of discussions, others felt that interactions were less authentic when peers relied heavily on AI, describing some posts as entirely AI generated.

  • Ease of Use: Students generally found M.S. Copilot intuitive to navigate and apply within the course context, suggesting minimal onboarding barriers.

Conclusion and Future Considerations

The study concludes that generative AI functions as a supplementary resource that students approached critically, often verifying and refining its output. The majority found it helpful in learning complex material, though concerns remain regarding accuracy, trust, and the potential erosion of independent reasoning.

The authors suggest that while generative AI can be valuable for exploration and reflection, its effectiveness depends on guided use: "These findings suggest that generative AI could potentially enrich advanced mathematics education when framed as a tool for exploration and reflection, but its effectiveness may depend on guided use that reinforces rather than replaces the process of writing and validating proofs. Future research is needed to identify best practices for integrating AI while preserving students’ ability to reason independently."

Improvements for AI systems

I. System Architecture Improvement: Implementing a Multi-Stage Procedural Grading Module (The Process Auditor)

The current limitation is that standard LLMs assess output, not process. The system must be enhanced with a dedicated module capable of tracking procedural compliance across multiple submission versions.

Specific Improvements:

  1. Constraint Graph Mapping: Implement a knowledge graph layer that maps specific assignment requirements (e.g., Must cite Theorem 3.2 from Chapter 4, Proof must use complete sentences) directly to verifiable tokens/phrases within the student's submission text and associated feedback comments.

  2. Differential Grading Layer: Develop a specialized comparison algorithm designed to quantify revision success (R success). This module accepts three inputs: (1) Draft 1, (2) Instructor Feedback Set (F), and (3) Draft 2. It calculates points based on the degree to which content in Draft 2 directly addresses and mitigates specific flagged deficiencies in F, rather than merely assessing the quality of Draft 2 independently.

  3. Formal Logic Checker for Proof Structure: Integrate a theorem prover (e.g., Lean or Coq backend) that operates over the natural language explanation. This module specifically flags structural fallacies like circular reasoning (where a premise relies on the conclusion being proven) or unjustified axioms, even if the final mathematical statement is correct.

What the Improved AI System Can Do:

The system can grade assignments based on a weighted function of Grade = w 1(Computational Correctness) + w 2(Procedural Compliance) + w 3(Revision Improvement Score). It moves beyond Is this correct? to "Did the student show they understood the required mathematical process, and did they improve based on expert critique?"

II. Module Enhancement: Contextual Justification Scorer (For Computational/Project Problems)

The rubric emphasizes that a final answer carries no points without adequate justification referencing class techniques. The AI must move beyond simple fact-checking to tracking the chain of reasoning.

III. System Enhancement: Longitudinal Engagement Tracker (For Discussion Forums)

The scoring logic for forum posts is complex and combinatorial (4 substantial posts times 2 days). This requires state-tracking over time, which current LLMs struggle to model accurately across multiple inputs.

Sources

Related papers