From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations

This paper investigates whether large language models (LLMs) can serve as auxiliary evidence sources for calibrating the difficulty of programming examinations in university courses, rather than merely being evaluated as code-generation targets. The study is motivated by the problem that "different teachers make differing judgments regarding knowledge-point coverage, algorithmic depth, implementation complexity, boundary data, problem statements, and time limits, which leaves student scores without a stable reference point in cross-class comparison, course-quality analysis, and instructional improvement."

The paper repositions LLMs from evaluation targets in code-generation benchmarks to auxiliary evidence sources for interpreting exam difficulty and builds a multi-evidence framework combining AI evidence, aggregated student performance, item exposure, online-judge process data, and teacher interpretation.

The research is organized around four research questions: (RQ1) whether synchronous AI answering data can form a relative difficulty ordering consistent with student pass rates in a real online judge environment; (RQ2) whether auditable API structured review can reflect student performance at exam and problem levels in parallel-class scenarios; (RQ3) how item-bank exposure, item-count pressure, non-attempt rate, and error-type distribution explain the applicability boundary of the AI difficulty ruler; and (RQ4) whether the auditable API review ruler supports longitudinal comparison of exam difficulty and problem risk across semesters and course levels.

The study uses four types of data. The first is a synchronous computer-based examination experiment in which 10 large language models and students answered the same exam problem set, with all code judged via OJ feedback. The exam contains 8 programming problems covering array processing, sorting, simulation, graph theory, dynamic programming, data-structure maintenance, and combinatorial counting, with 120 students as statistical objects. The second dataset comes from the final computer-based examinations of 11 parallel classes in the same semester: a total of 79 programming problems, with each exam having problem statements, reference answers, and class-wide whole-exam score distributions. The third dataset comprises 26 problems across 4 final computer-based examinations of the same teacher's Data Structures and Algorithms B (Spring 2024, Spring 2025, Fall 2025, and Spring 2026). The fourth dataset consists of 106 problems across 16 past final computer-based examinations of the same teacher's Introduction to Computing B (Fall 2008 through Fall 2025).

The paper distinguishes two forms of the AI ruler. The first is the solving-based AI ruler, where the AI reads the problem statement, generates code, submits it to the OJ, and performs a limited number of revision rounds based on judging feedback, producing process data such as pass rate, number of attempts, running time, and error type. The second is the review-based AI ruler, where "the AI does not submit code but, based on the problem statement, constraints, and reference answer, provides structured scores for concept difficulty, implementation difficulty, debugging difficulty, complexity risk, statement-reading difficulty, and overall difficulty" on a 1-to-5 scale.

For the solving-based experiment, the paper constructs a composite difficulty index: dp = α(1 − ΣAC/M) + β·min(1, Attempts/5) + γ·min(1, Time/TL), with α=0.6, β=0.2, γ=0.2. The weights are justified by the pass rate being the most direct proxy indicator of difficulty while the number of attempts reflects the process cost of solving, and the running-time ratio reflects computational-efficiency pressure.

For the review-based calibration, the paper uses a single-reviewer design whose historical selection basis comes from the first-stage solving experiment: among the 10 participating models, ChatGPT was the only model that passed all 8 problems (AK) and cracked the extremely difficult problem I30547. However, the paper explicitly discloses that "the review batch actually used in this paper is invoked through a third-party OpenAI-compatible endpoint, and the model identifier in both the request and the endpoint's return is gpt-5.6-sol; this label cannot certify that the upstream is an official OpenAI model. The three review batches (79 problems, 26 problems, 106 problems) were run on July 10, 2026 under a unified protocol: temperature=0, reasoning effort=high, and JSON-schema-constrained output, with complete provenance metadata including the model identifier, invocation date, endpoint, temperature, the SHA-256 of the prompt and schema, the raw API response, and run metadata."

The paper also introduces an item-bank exposure variable, distinguishing intrinsic difficulty from effective exam difficulty. For the 12 practice item-bank original problems, the paper adopts a conservative discount discount p=0.25, with sensitivity analysis across discount levels 0, 0.10, 0.25, and 0.40.

The first-stage results show that "the Spearman rank correlation between AI pass rate and student problem pass rate is 0.866, with an exact two-sided permutation p=0.0119; the Spearman rank correlation between the composite difficulty index dp and student problem pass rate is-0.905, with an exact two-sided permutation p=0.0046. The paper notes that the composite index exhibits stronger ranking consistency than pass rate alone, indicating that the number of attempts and time cost during solving carry additional explanatory value. The AI solving performance formed a clear gradient: basic problems were passed by all or nearly all models, while the extremely hard problem was passed by only 1 model (ChatGPT, which solved all 8 problems). The paper also notes absolute-level differences: on medium-high-difficulty problems such as T30913, M30947, and U30919, the AI pass rate is markedly higher than the student pass rate, indicating that the AI scale is suitable for relative difficulty ranking and problem-setting review, but not for directly predicting the proportion of students who pass."

The cross-sectional review-based calibration results show that the 11 parallel-class exams differ in item count, high-difficulty ratio, and item-bank exposure. The overall difficulty of the 79 problems covers levels 1 to 5 (13/26/25/10/5 problems). At the problem level, "the Spearman correlation between raw AI overall difficulty and student pass rate is-0.871 (p<0.001, Fisher-approximate 95% CI [-0.916, -0.805]), and-0.857 after exposure adjustment. After removing the 12 item-bank original problems, the Spearman correlation between raw AI difficulty and pass rate is still-0.866." The correlation between raw AI difficulty and non-attempt rate is 0.800. At the exam level, the correlations are weaker: across all 11 exams, the Spearman correlation between adjusted AI difficulty and normalized student mean is-0.461 (Pearson r=-0.598), which can only be interpreted as exploratory evidence.

The sensitivity analysis of the exposure discount shows that the direction and strength of the problem-level correlation are generally stable: across the four discount levels, the Spearman ρ between adjusted AI difficulty and pass rate ranges from-0.780 to-0.871. Notably, "the correlation is strongest when the discount is 0 (i.e., no adjustment), and it weakens monotonically as the discount increases; this is contrary to the assumption that 'the exposure discount improves explanatory power.' The paper concludes that the exposure discount can therefore only serve as a contextualizing explanatory parameter, not a causal estimate of student performance."

The error-type analysis shows that among the 79 problems, the dominant error type is Wrong Answer for 65 problems, Runtime Error for 8 problems, Time Limit Exceeded for 4 problems, and Compile Error for 2 problems. The Spearman correlation between the AI complexity-risk score and the student TLE-submission share is 0.350, the strongest relationship with TLE among the six review dimensions, though its strength is only moderate.

The longitudinal Data Structures and Algorithms B results show that the mean AI difficulty rises semester by semester from 2.000 in Spring 2024 to 3.375 in Spring 2026, and the high-difficulty ratio rises from 0 to 0.500. On the item-level data of 26 items, overall AI difficulty is strongly negatively correlated with student pass rate (Pearson r=-0.795, Spearman ρ=-0.829) and strongly positively correlated with the non-attempt rate (Pearson r=0.848, Spearman ρ=0.883).

The Introduction to Computing B sample marks the boundary of the scale: the overall difficulty of the 106 items covers levels 1 to 5 (8/41/31/24/2 items), and 26 items are rated as high-difficulty, but "at the item level the Pearson/Spearman between overall AI difficulty and student pass rate is-0.550/-0.552, and with the non-attempt rate is 0.541/0.564—the correlation direction is correct and the strength is moderate, clearly weaker than the-0.86/-0.83 magnitude of the two Data Structures and Algorithms B samples. At the exam-paper level across the 16 exams, the Spearman correlation between mean AI difficulty and mean student pass rate is only-0.063 (Pearson r=-0.168), close to zero. The paper explains that cohort differences rather than item difficulty dominate exam-paper-level performance," citing the contrast between domestic and international student papers of the same year.

The paper presents four illustrative cases: Class 16 shows high AI difficulty and low student performance with a normalized mean student performance of 0.191 and low-performance rate of 0.636; Class 15 shows that a paper with the highest AI difficulty (3.375) can still maintain a relatively high student completion level (mean 3.817 problems passed, low-performance rate 0.067) due to structural division of labor between 'basic coverage' and 'discriminating top students'; Class 9 with 13 items shows that item count, the switching cost between items, the reading burden, and the pressure of time allocation themselves constitute sources of difficulty at the exam-paper level; and the 12 item-bank original items across three papers illustrate that whether an item has been seen before and whether it comes from an in-course practice system will change the effective difficulty and the evaluative meaning of the paper.

The paper's conclusions emphasize that "the AI difficulty ruler is only suitable for serving item-setting review, exam-structure analysis, problem-level risk diagnosis, parallel-class fairness discussion, longitudinal quality tracking, and benchmark construction; it cannot directly predict individual student performance, cannot replace teachers' professional judgment, and still less can automatically trigger score adjustment. The validity of the evidence is bounded by three source limits: the review is completed via a third-party OpenAI-compatible endpoint, and the model label cannot authenticate the upstream as an official OpenAI model; the reviewer is a single reviewer, and there is as yet no source-complete multi-reviewer reliability; the repeated-problem test shows that temperature=0 does not constitute a guarantee of output determinism, but the scoring jitter does not change the main correlations."

The paper also discusses ethical boundaries and governance requirements, including that AI results can only be used for auxiliary analysis at the problem and exam levels, and cannot be directly used to evaluate individual student ability, that student data should be limited to anonymized group statistics, that the model version, prompt version, review date, endpoint, temperature, prompt hash, and scoring schema must be recorded together with the results, that the AI ruler should not automatically trigger score adjustment, and that students should enjoy the right to be informed regarding AI-assisted evaluation.

Limitations acknowledged include: the solving-based experiment has only 8 problems and 10 AI models; the review-based evidence comes from a single batch run of a single reviewer via a third-party endpoint; the first author's insider expert knowledge as the instructor of Class 15 may bring item-setter perspective bias; the nested structure of 79 problems within 11 exams limits statistical inference; the analysis lacks finer process indicators such as student ability stratification and complete submission sequences; the whole-exam student performance mixes two statistical calibers; the exposure discount is a conservative empirical setting with exposed problems concentrated in Class 8; there are slight differences between ranking-page statistics and source-document aggregated buckets; and there is a potential training-data contamination problem, though the paper notes mitigating factors including that "the 10 models in the first stage answered in the same environment, and if ChatGPT's memory advantage were caused by contamination, it would be hard to explain why the other models did not reach the same level on all problems."

Future research directions include expanding problem-level student process data, introducing multiple AI reviewers and teacher-expert scoring, expanding to multi-semester multi-course multi-teacher samples with hierarchical or IRT models, and building an AI-calibration benchmark for educational-scenario programming exams that incorporates parallel-class exam problems, longitudinal exam problems, problem-exposure annotations, AI answering-process data, AI review data, student-group statistics, online-judge process data, and external code-benchmark anchors.

Based on the paper, here are specific improvements that can be made to AI systems, along with what the improved systems can do:

• Improvement: Train or fine-tune models to output structured, multi-dimensional difficulty scores (concept, implementation, debugging, complexity risk, reading difficulty, overall) for programming problems, not just solve them.

• What the improved system can do: Given a problem statement, constraints, and reference solution, it can produce a 1–5 scale difficulty profile that correlates with real student pass rates (Spearman ρ = −0.871) and non-attempt rates (ρ = 0.800) at the problem level, enabling teachers to pre-screen exam items.

• Improvement: Extend the evaluation loop so that after generating code, the system records: pass/fail per problem, number of repair attempts, and runtime-to-time-limit ratio. Then compute a composite difficulty index (e.g., α=0.6 pass rate, β=0.2 attempts, γ=0.2 time ratio).

• What the improved system can do: Produce a relative difficulty ordering of exam problems that matches student performance (Spearman ρ = −0.905), even when pass rates alone are not discriminative (e.g., problems with high pass rate but near-time-limit runtime).

• Improvement: Integrate an item-bank exposure flag into the review input, and apply a configurable discount (0.00–0.40) to the raw difficulty score to estimate effective exam difficulty.

• What the improved system can do: Distinguish intrinsic difficulty from effective difficulty in contexts where students have seen the problem before, and flag cases where exposure does not reduce difficulty (as observed in this study), preventing over-adjustment.

• Improvement: Wrap the review call so that every request stores: endpoint, temperature, prompt/schema SHA-256 hashes, raw response, run metadata, and model identifier. Add a validation step that cross-checks normalized output against raw responses and hashes.

• What the improved system can do: Provide fully auditable, reproducible difficulty reviews that can be verified by third parties, and detect when a script or non-AI process is mislabeled as AI output (a failure mode identified in the paper).

• Improvement: Train the reviewer to predict, for each problem, the dominant student error type (WA, TLE, RE, CE) and the expected proportion of erroneous submissions, based on the problem’s complexity risk and implementation difficulty dimensions.

• What the improved system can do: Alert teachers to specific risk mechanisms (e.g., high TLE risk for naive algorithms, high RE risk for boundary handling) before the exam, enabling targeted teaching interventions.

• Improvement: Implement a warning system that detects when the AI difficulty scale is being used at the exam level in courses where cohort composition dominates performance (e.g., introductory courses with international vs. domestic students).

• What the improved system can do: Automatically flag when exam-level difficulty comparisons are invalid (e.g., near-zero correlation with student performance), preventing automatic grade adjustments or unfair cross-class comparisons.

• Improvement: When the same problem statement and reference code are submitted multiple times, the system should report the variance in overall difficulty scores (e.g., ±1 level on 5 of 13 duplicate groups in this study).

• What the improved system can do: Quantify its own output instability, allowing users to treat single-reviewer scores as exploratory rather than deterministic, and to compute confidence intervals for difficulty estimates.

• Improvement: Extend the review pipeline to accept student stratification data (e.g., by prior OJ experience, course-selection background) and output difficulty scores per subgroup.

• What the improved system can do: Detect when the same problem presents different difficulty to different student groups, enabling discussion of within-class fairness—a capability the paper explicitly identifies as missing.

The improved system can serve as a source-auditable, problem-level difficulty calibration tool that:

• Pre-screens exam items with multi-dimensional difficulty scores that correlate with real student performance.

• Provides a solving-based difficulty index that captures pass rate, attempt cost, and time pressure.

• Adjusts for item-bank exposure with a configurable discount.

• Maintains full provenance for every review call, preventing misattribution.

• Predicts error-type mechanisms and flags complexity risks.

• Detects when its own scale is invalid at the exam level (e.g., in introductory courses).

• Reports its own output instability on duplicate inputs.

• Can be extended to subgroup-level analysis for within-class fairness.

It cannot, and should not, be used to evaluate individual students, automatically adjust grades, or replace teacher judgment.