FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays. Jane, what’s the big picture here?

Jane: Well, basically, it’s about comparing how human writing instructors give feedback to argumentative essays versus what four different frontier Large Language Models produce when they are given the same instructions. It sets up a system to see where the human and model feedback lines up and where they totally drift apart in terms of what's important in revision.

Lu: I think it’s significant because for writing research, we always talk about three key things: goal-orientation, anchoring to specific sentences, and prioritization. This paper is the first one systematically comparing how instructors and models handle those exact three dimensions across a whole dataset.

Meng: So it’s not just about whether the feedback is good or bad; it’s about *how* they focus on the essay structure itself, right? Like, which parts of the argument are they actually hitting when they give advice?

Tom: Exactly. The authors built this dataset called FOXGLOVE, which has a total of two thousand three hundred forty comments—six hundred ninety-six from human instructors and one thousand six hundred forty-four from four different AI models—all using the same set of rules <ref:2606.06271#pg1>.

Jane: And what they found is that while both groups generally agree on which argumentative goals are important across essays and positions, they really diverge when it comes to *which specific sentences* they pick to comment on and how urgent those comments are.

Lu: I also saw something interesting about the models themselves; the paper notes that LLMs tend to write feedback that is more complex than instructors do, and they use fewer questions overall.

Meng: That’s a big point for deployment because complexity can be good, but if it’s not actionable or focused on what actually needs fixing, it just adds noise at scale.

Tom: And the quality ratings are a bit nuanced; LLM feedback often gets higher scores on most quality dimensions from the instructors, but the paper immediately points out that this advantage is mostly because their comments are longer.

Jane: So even when they agree on what’s important, we have to be careful about trusting the model just because it wrote a longer piece of text. That leads us right into how they actually constructed this comparison.

The paper's summary: Tom: Now that we know what FOXGLOVE is, let's talk about the core methodology described in the paper. Basically, they’re using argumentative essays from a corpus called PERSUADE two point zero—which consists of high-school student writing—and annotating every sentence with five specific goals: Position, Claim, Evidence, Counterclaim, and Rebuttal <ref:2606.06271#pg2>.

Jane: That sentence-level labeling is crucial because it lets them measure feedback directly against those five rhetorical goals for each part of the essay. It’s not just a general comment; it’s tied to a specific argumentative task.

Lu: They also established a very specific feedback schema that everyone, both humans and models, had to follow. Every piece of feedback has three parts: sentence-level comments anchored to the text, an urgency ranking where you pick the top three most important ones—ranked one being most urgent—and a global comment summarizing how the essay fits the goals.

Meng: That shared protocol is what makes it a true comparison; if they weren't all following the same rules, we wouldn't know if their differences were due to different training or just different ways of thinking about the task.

Tom: Right, and this setup allowed them to systematically compare where instructors and models align on those goals versus where they diverge on sentence selection and prioritization.

Jane: The summary emphasizes that even though the distribution of feedback across goals seems similar between humans and AI, the specific choice of text—the anchoring—and the urgency ranking are where the real differences emerge.

Lu: It’s interesting because they found that models tend to write more complex feedback and use fewer questions than instructors when we look at those sentence-level comments.

Meng: That makes sense if the model is trying to be thorough, but Tom mentioned earlier that this length can actually hurt quality scores unless it translates into better specificity.

Tom: It does. The paper sets up this comparison so we can see if that extra length in the AI feedback actually leads to better results or just looks good on paper because it’s longer.

The paper's improvements: Tom: So, the authors didn't just stop at showing the differences; they suggested ways to make their work better, and those improvements are pretty interesting for how we use this kind of data.

Jane: The first improvement they suggest is to fix the issue with models plateauing on certain types of text spans. They found that GPT-five point two and Qwen3-Next 80B kept failing consistently on a few specific sentence highlighting cases, which suggests we need better refinement loops than just simple resamplings <ref:2606.06271#pg2>.

Lu: And the second improvement is directly addressing the length bias we talked about earlier. They pointed out that longer comments correlate with higher scores in specificity and actionability for both groups, so they propose a way to control the output length to be closer to a human average of around sixty-four words <ref:2606.06271#pg1>.

Meng: Controlling the length is smart because it removes that confounding variable where the models look better just because they write more text. If we force them toward that average, we get a fairer comparison of quality versus quantity.

Tom: And the third suggestion is about testing if eliciting more varied feedback actually helps usefulness, rather than just optimizing for expert ratings alone. They want to see if variation can be a strength or just noise.

Jane: That’s a really important direction for future research because it moves beyond just getting the highest possible score and asks whether different styles of feedback actually serve different revision needs.

Lu: I think those suggestions give us concrete protocols we can use in the future to test if we can actually guide models toward producing feedback that is more focused on what matters for the student’s learning process.

Meng: If they get a method that forces length control, that’s something my team could start prototyping right away to make sure our deployed models aren't just outputting verbose nonsense.

Conclusion: Tom: So wrapping up the FOXGLOVE paper, the main implication is that we can finally have a systematic way to compare where human and AI feedback agree and disagree on the core elements of revision: what goals to target, which specific text spans matter, and how urgent those comments are.

Jane: It shows that while AI can generate comprehensive feedback in terms of length, human judgment still remains essential for deciding the precise focus needed for effective revision.

Lu: The paper gives us reusable protocols that could be used in writing-intensive contexts to test if we can connect rated quality directly to actual improvements in student writing outcomes.

Meng: For deployment, it suggests that we need methods to manage the output style so we aren't just relying on length as a proxy for usefulness when integrating AI feedback into workflows.

Lalam: From my perspective, FOXGLOVE confirms that the models are very good at covering all five goals in their global comments, but they still struggle with the fine-grained anchoring to the exact sentences that matter for a specific claim or evidence point.