Who Gets Heeded? An Obligation-Level Audit of Responsiveness in EPA Rulemaking

This paper introduces obligation-level responsiveness auditing, an auditable, AI-assisted framework for measuring whether public-comment engagement co-occurs with changes to specific regulatory duties in U.S. notice-and-comment rulemaking. The framework extracts obligations from proposed and final-rule text, matches comments to the obligations they address, and classifies proposed-final outcomes, with each load-bearing component evaluated against blind human judgment.

The authors assembled a population-scale corpus of 786,197 EPA public comments across 6,145 rulemaking dockets from 2010–2022, selected 36 anchor rulemakings via stratified random and extreme-case sampling, and constructed a 70,075-comment analytic sample within those anchors. The obligation-extraction pipeline produced 12,730 obligations across 29 productive anchor rulemakings, with the obligation-comment matcher producing 398,509 comment-obligation pairs, of which 18,266 pairs (4.58%) carried addressed=True, and 684 distinct obligations (5.6%) were addressed by at least one commenter.

Three descriptive empirical findings emerge:

Finding 1: Engagement is associated with revision, but modestly. Obligations addressed by five or more commenters were 19.5 percentage points more likely to be revised between proposed and final than less-addressed obligations (69.0% vs 49.5%; chi2 = 24.53, p < 0.001, Cramér’s V = 0.044, n = 12,730). A docket-fixed-effects logistic regression with cluster-robust standard errors recovered an interpretable within-docket effect: high-engagement obligations have nearly twice the within-docket odds of revision (OR = 1.93, 95% CI [1.04, 3.57], p = 0.037; 29 docket clusters). The pattern is real and modest; most rule revisions occur on obligations that no commenter addressed, and most addressed obligations are not revised.

Finding 2: Direction of engagement does not differentiate outcomes, a substantive null. Among obligations addressed by at least one opposing commenter (n = 212), 62.3% were revised; among those addressed by at least one supporting commenter (n = 312), 68.3% were revised; the difference is not statistically significant (two-proportion z = −1.42, p = 0.155). With these sample sizes the analysis would be unlikely to miss a large directional difference, so the null is interpreted as substantively informative rather than a power deficit.

Finding 3: Under a permissive reconstruction of commenter type, organizational-majority engagement is concentrated in editorial-refinement outcomes at the cross-docket level. On the 116 engaged obligations with a SURVIVED-edited or MODIFIED outcome, the audit-corrected 2×2 contingency yields cells (SE-org=13, SE-not-org=2, MO-org=57, MO-not-org=43) with Fisher’s exact OR = 4.90, p = 0.044, n = 115. The audit-corrected SURVIVED-edited bucket (n = 15) is 87% organizational-majority; the audit-corrected MODIFIED bucket (n = 100) is 57%. The pre-audit classifier-assigned contingency was cells (43, 15, 28, 30), OR = 3.07, 95% CI [1.41, 6.71], p = 0.007 (n = 116). A docket-clustered logistic specification confirms the cross-docket result (OR = 3.07, 95% CI [1.08, 8.72], p = 0.035, 15 docket clusters). The within-docket Cochran-Mantel-Haenszel common odds ratio is statistically inconclusive (CMH OR = 1.42, 95% CI [0.51, 4.00], p = 0.507).

A blind dual-annotator audit of the outcome-state classifier revealed that the cosine-similarity classifier agrees with human raters at only kappa = 0.137 on the SURVIVED-edited/MODIFIED contrast, establishing that text-surface similarity is a brittle proxy for substantive compliance change at this boundary. The audit preserved the direction of the equity disparity under corrected labels with a larger point estimate (audit-corrected Fisher’s exact OR = 4.90, p = 0.044, n = 115), while reducing statistical precision.

The paper's contributions are threefold: (1) a conceptual/measurement contribution arguing that the relevant unit for auditing administrative responsiveness is the discrete regulatory obligation, not the rule, docket, or comment; (2) an empirical contribution applying the framework to a population-scale EPA corpus yielding the three descriptive findings; and (3) a methodological contribution developing and validating an auditable, LLM-assisted pipeline for obligation extraction and obligation-comment matching at population scale on legal-policy text.

The authors conclude that the equity asymmetry is located upstream of agency response: in differential capacity across commenter populations to identify, interpret, and contest specific legal obligations. Formal openness in notice-and-comment rulemaking can mask upstream structural inequity, and obligation-level analysis makes this pattern visible.

Improvements to AI systems:

1. Obligation-Aware Text Matching for Regulatory Documents

• Build an AI system that extracts discrete legal obligations (duties, prohibitions, permissions) from proposed and final rules, rather than treating whole documents as units.

• The system can link public comments to specific obligations via semantic matching, enabling fine-grained tracking of which duties are addressed, revised, or ignored.

• Capability: Automatically generate obligation-level responsiveness reports for any rulemaking docket, flagging high-engagement obligations that were not revised.

2. Audit-Corrected Outcome Classification

• Replace brittle cosine-similarity classifiers with a hybrid human-in-the-loop pipeline that detects when text-surface similarity fails to capture substantive compliance change (e.g., SURVIVED-edited vs. MODIFIED).

• The system can flag low-confidence boundary cases for blind human audit, then recalibrate labels using audit-corrected ground truth.

• Capability: Produce reliable revision-outcome labels (unchanged, edited, modified, removed) for regulatory obligations, with uncertainty estimates and audit trails.

3. Directional Engagement Analysis with Null-Aware Reporting

• Implement statistical tests that explicitly distinguish “no effect” from “insufficient power,” using pre-registered sample-size calculations and equivalence testing.

• The system can report whether comment direction (support vs. oppose) has no detectable association with revision, and quantify the minimum detectable effect size.

• Capability: Generate transparent, decision-ready summaries of whether public sentiment direction influences regulatory change, avoiding false claims of influence.

4. Equity Disparity Detection via Commenter-Type Inference

• Train a classifier to infer commenter type (organizational vs. individual) from text and metadata, with permissive reconstruction and audit correction.

• The system can compute odds ratios (e.g., Fisher’s exact, CMH) for organizational-majority engagement across outcome categories, stratified by docket.

• Capability: Automatically flag rulemakings where organizational comments disproportionately drive editorial refinements, revealing upstream capacity inequities.

5. Population-Scale Comment-Obligation Pairing

• Scale the matching pipeline to millions of comments and thousands of dockets using LLM-based retrieval and deduplication, with precision/recall validated against blind human judgment.

• The system can process entire EPA (or other agency) corpora in hours, producing obligation-level engagement matrices.

• Capability: Enable longitudinal audits of regulatory responsiveness across agencies, decades, and policy domains, identifying systemic patterns of neglect or capture.

6. Blind Dual-Annotator Audit Integration

• Build an AI system that automatically selects a stratified random sample of classifier outputs for blind dual-annotator review, computes inter-rater reliability (e.g., Cohen’s κ), and adjusts downstream statistics accordingly.

• The system can detect when a classifier’s performance degrades at specific boundaries (e.g., SURVIVED-edited vs. MODIFIED) and trigger retraining or fallback to human coding.

• Capability: Continuously self-auditing AI pipelines for legal-policy analysis, with published audit metrics and corrected effect sizes.

7. Within-Docket vs. Cross-Docket Effect Separation

• Implement hierarchical models (e.g., docket-fixed-effects logistic regression, CMH tests) that separate within-docket from cross-docket effects, preventing Simpson’s paradox.

• The system can report both pooled and stratified odds ratios, with cluster-robust standard errors.

• Capability: Distinguish whether engagement effects are driven by docket-level confounders or by obligation-level dynamics, improving causal interpretability.

8. Obligation-Level Responsiveness Dashboard

• Create an interactive tool that visualizes, per obligation: number of addressing comments, direction mix, commenter-type majority, revision outcome, and audit confidence.

• The system can rank obligations by “engagement-revision gap” to prioritize agency review.

• Capability: Empower regulators, journalists, and civil society to monitor specific regulatory duties in real time, exposing silent revisions or ignored high-engagement duties.

Notice-and-comment rulemaking gives any affected party the same formal right to influence federal regulation, but formal access is not substantive capacity to shape rule text. Existing strategies operate at the rule or aggregate-corpus level, too coarse to capture the discrete regulatory obligations where commenters seek change. We introduce obligation-level responsiveness auditing, an auditable, AI-assisted framework for measuring whether public-comment engagement co-occurs with changes to specific regulatory duties. The framework extracts proposed and final-rule obligations, matches comments to the obligations they address, and classifies proposed-final outcomes; each load-bearing component is evaluated against blind human judgment. We apply the framework to 70,075 comments across 36 EPA anchor rulemakings, drawn from a corpus of 786,197 comments across 6,145 dockets from 2010-2022. Three descriptive findings emerge. First, engagement is associated with revision at a modest within-docket magnitude. Second, support-versus-opposition direction does not clearly differentiate outcomes, an informative null inconsistent with simple preference-aggregation. Third, under a permissive reconstruction of commenter type, organizational-majority engagement concentrates in editorial-refinement rather than substantive-modification outcomes at the cross-docket level. A blind human audit of the load-bearing outcome contrast preserves this third finding under corrected labels and reveals that text-similarity methods are insufficient for distinguishing editorial from substantive regulatory change, a measurement-validity lesson we treat as a supporting methodological contribution. Together, these findings locate the equity asymmetry upstream of agency response: in differential capacity across commenter populations to identify, interpret, and contest specific legal obligations.