Authoring and Management of Transparent Research Integrity Assessments of Randomised Clinical Trial Publications Using LLM-assisted Tools and Provenance Knowledge Graphs

arXiv:2608.07202 · cs.AI · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Authoring and Management of Transparent Research Integrity Assessments of Randomised Clinical Trial Publications Using LLM-assisted Tools and Provenance Knowledge Graphs".

Jane: The paper was written by Milan Markovic, Goutham Indukuri, Somayajulu Sripada, Colby J. Vorland, Jack Wilkinson et al. from University of Aberdeen and Indiana University and University of Manchester and University of Auckland.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper Summary: Tom: We've got a fascinating paper to open with today, all about keeping unreliable clinical trial results from quietly leaking into medical practice. The author list runs across computing science, epidemiology, and biostatistics, which tells you straight away this is a practical problem. And the whole thing hangs together as one pipeline.

Jane: It really is a systems story. There's an eye-assisted assessment tool, a formal ontology for recording how each assessment happened, and a public knowledge graph holding the results. Those three pieces make up the RIPE Observatory.

Lu: The problem they're tackling is serious. Systematic reviews of randomized trials feed directly into clinical guidelines, so a falsified trial doesn't just waste money, it can change what doctors recommend. That's the stakes for patients.

Tom: The paper cites earlier work showing 27 trials with integrity concerns had found their way into 88 systematic reviews or clinical guidelines, changing findings in over half of them. In 87 percent of those cases, the change was substantial enough to shift the direction of effect.

Meng: That's a scary number to open with.

Tom: It is. And it's exactly the kind of downstream harm that makes integrity assessment more than an academic exercise.

Meng: Which is why they built INSPECT-AI. It walks a human reviewer through the INSPECT-SR checklist, the Cochrane-endorsed integrity framework, while an LLM pulls evidence from the PDF and from external registries and databases. Every suggestion has to be confirmed or overridden by a person.

Jane: And the clever bit is the provenance side. Every piece of evidence, every automated suggestion, every override gets recorded. You can trace why a trial was flagged, step by step.

Lalam: That's the piece that's been missing everywhere. There are integrity checklists, and there are eye assistants, but nobody has been publishing the full audit trail of the assessment in a machine-readable, FAIR-aligned form.

Tom: And the released dataset is substantial — 140 assessments of 95 trial publications sitting in an open knowledge graph with a SPARQL endpoint.

Lu: But the results also show why the human in the loop matters. Automated suggestions and human reviewers disagreed on about 14 percent of individual check outcomes. And on trial registration questions, human reviewers disagreed with each other more than half the time.

Jane: That sounds like a weakness at first, but it's actually the paper's strongest argument. You need the provenance precisely because the humans disagree, and because the eye disagrees with the humans.

Meng: And the cost numbers make it feasible. The whole pilot ran on less than eighty dollars, with the language model portion just over a dollar and each paper coming in below ten cents.

Lalam: That combination — rigorous provenance plus trivial cost — makes me think this is a template for how any evidence-based field could audit its literature.

Tom: The abstract already tells you how carefully they've positioned this work. Let's look at that opening page.

Page 1 of the Paper: Tom: So we've sketched the whole arc, and now we can read the opening page the way the authors intended. The abstract's first sentence describes INSPECT-eye as an interactive tool, and that word carries real weight. The system proposes, but a person confirms or overrides every decision before it counts.

Jane: It also says the INSPECT-SR framework is community approved, which is a crucial detail. That checklist went through a formal consensus process with international integrity researchers before Cochrane endorsed it. You can't claim that for most instruments in this space.

Lu: The abstract ties the tool to the honesty principle in research publishing. There are several principles of research integrity, but this work deliberately focuses on one question: are the data and findings trustworthy enough for evidence synthesis?

Meng: And then the scale: 140 expert assessments of 95 publications. Those assessments were done by real reviewers using the tool, and each one is captured in the ontology's structure. The word "initial" matters too — this is a starting point, not a finished corpus.

Lalam: What I find telling is the permanent identifier at the bottom — w3id.org/ripe. That's infrastructure thinking. The authors are saying this resource is meant to persist and be referenced, not to live and die with a grant cycle.

Lu: And that's rare for a research prototype.

Lalam: Exactly. The choice of a stable web identifier signals intent from day one.

Jane: The author list tells the same story. Computing scientists from Aberdeen sit alongside epidemiologists and biostatisticians from Indiana, Manchester, and Auckland. That mix explains why the human workflow and the technical machinery stay in view together.

Tom: The keywords are equally revealing — research integrity, large language model, ontology, provenance, knowledge graph. That's a signal to two communities at once. The integrity world gets a tool, and the semantic web world gets a model it can build on.

Meng: One phrase in the abstract deserves attention too. It says the assessments were "described using RIPE-O," which implies the ontology came after the assessments, shaped by what actually happened. That's the opposite of a top-down modeling exercise.

Jane: From that framing, the paper moves to laying out the RIPE Observatory itself — three named components, each with a permanent identifier. That's where the architecture gets concrete.

Page 2 of the Paper (Discussing Page 3): Tom: The next page introduces the RIPE Observatory as a collection of semantic resources, and each component gets its own permanent identifier — the tool, the ontology, and the knowledge graph. That's the architecture made explicit.

Jane: There's a refreshingly honest division of labor here. They name the ontology and the knowledge graph as the core semantic contributions, while the tool is included because it shaped them and completes the pipeline. But they also say the ontology is meant to work with other integrity tools, not just theirs.

Meng: That openness is the difference between a project deliverable and actual infrastructure. There are other checklists in this space — TRACT, RIA, REAPPRAISED, and the tool used in Cochrane pregnancy research. If each one produces its own incompatible data format, you haven't solved the problem.

Lu: The related work section draws the line against the existing scholarly knowledge graphs. OpenAIRE, Scholia, Semantic Scholar, the Open Research Knowledge Graph, SemOpenAlex — they all integrate enormous amounts of bibliographic data.

Jane: But none of them carry data about trustworthiness. No retraction records, no expressions of concern, no integrity flags. That gap is what this paper claims.

Tom: And the authors point out that discovering problematic trials today is still a manual slog. Integrity sleuths alert journals, but publishers can be slow to act, and a high proportion of untrustworthy publications remain unflagged years later.

Jane: Which seems backwards for a world that tracks citations in real time.

Tom: Completely. The scholarly web knows more about who cites whom than about whether a study deserves to be cited at all.

Meng: What they're adding is the layer that records the checks being performed, in a form computers can reason over. That's a different kind of contribution from another checklist.

Lalam: And they're aware of adjacent attempts, including an early experiment semi-automating the TRACT checklist with GPT-4o. So they're positioning themselves against known work rather than ignoring it.

Lu: The paragraph about the checklists also explains why INSPECT-SR matters — it's the one endorsed by Cochrane, which gives the assessments institutional weight from the start.

Jane: Having situated themselves against the field, the next section explains how they actually built the system. And the methodology is as interesting as the results.

Page 3 of the Paper (Discussing Page 5): Tom: The methodology section reads like a case study in interdisciplinary tool building, and the biggest outcome is the decision to keep a human in the loop from day one. That decision emerged from the development process rather than being a default starting point.

Jane: They're open about how it happened. The non-technical team members understood the strengths and limits of eye only after seeing real, semi-functional prototypes. Paper mock-ups and technical explanations didn't land the same way.

Lu: And this is where having biomedical researchers in the room mattered. They supplied fifty real problematic publications with detailed guidance on how to assess them. That gave the computing team the domain grounding no textbook could provide.

Meng: The order of operations is also surprising. They built the tool first, and the ontology afterwards, derived from the logs the running application actually produced. Most ontology work starts with a model and then hunts for data to fit it.

Jane: That's the reverse of the usual pattern.

Meng: It is. And it's probably why the model feels grounded in real assessment practice rather than abstract theorizing.

Lalam: The provenance graphs were generated from those JSON logs using declarative YARRRML mappings. So the audit trail is a byproduct of the system in use, not a reconstruction attempted after the fact.

Jane: For the ontology itself they followed a formal engineering framework called LOT, with competency questions framed around the seven Ws of provenance — who, what, where, which, why, when, how.

Tom: And the paper makes a smart reuse decision by building on TIDO, an ontology from threat intelligence. That brings a forensic framing with it — an integrity assessment is treated like an investigation, with evidence, hypotheses, and evaluation activities.

Meng: TIDO already aligns with the W3C provenance standard, PROV-O, so all of this inherits standard semantics. Interoperability comes almost for free.

Lu: They also reuse the SPAR ontologies for bibliographic details rather than reinventing how to describe authors and publications. That's the kind of hygiene that makes an ontology adoptable beyond the team that built it.

Jane: All of that design work then got exercised in a real pilot with real reviewers. And the numbers from that pilot are surprisingly good.

Page 4 of the Paper (Discussing Page 7): Tom: The pilot deployment is a genuine field test. Thirteen volunteers from three universities spent October and November last year using the tool on real publications.

Jane: And these weren't random volunteers. 77 percent had experience conducting systematic reviews, and nearly 39 percent had ten or more years of that experience. When people like that say a tool is usable, it carries weight.

Lu: They assessed 69 publications and produced 104 assessment traces, with some papers assessed by multiple reviewers. That overlap is exactly what you need if you want to study disagreement later.

Meng: The usability results were strong. Nobody found the tool difficult to use, 69 percent called it user-friendly, and 69 percent rated the response time as quick or very quick.

Tom: And no negative ratings on any of those scales.

Meng: Right — zero difficulty complaints. That's a clean result for a prototype.

Lalam: One detail I like is that participants were given publications with known trustworthiness issues, but they were also encouraged to choose their own. The traces reflect genuine reviewer behavior, not a scripted exercise.

Tom: The paper is also honest that the tool didn't remove the complexity of integrity assessment. It made the process faster and better supported, which is a more realistic promise than total automation.

Jane: There's a subtle point about who these reviewers were. Only 61 point 5 percent had done integrity assessments before, although 77 percent had done risk of bias assessments. Related skills, but not the same thing.

Lu: That actually strengthens the pilot. If health researchers with mixed backgrounds can pick up the tool and produce assessments, that's decent evidence for wider adoption.

Meng: And the data from this pilot became the raw material for the knowledge graph. The volunteer traces, combined with assessments by the expert core team, grew into the 140 published assessments.

Jane: With that data in hand, the paper moves to the ontology. And the model turns out to be a lot more generic than you'd expect from a tool built around one checklist.

Page 5 of the Paper (Discussing Page 9): Tom: The ontology section opens with a design claim that's easy to miss but really important: RIPE-O doesn't prescribe which questions are asked. It models the structure of an assessment, not the content of any particular checklist.

Jane: So a question like "are there concerns about the timing of study registration" becomes an integrity assessment question. Evidence is gathered, an evaluation activity weighs it, and the answer is recorded as a hypothesis with an outcome.

Meng: The consequence is that the same structure could document integrity concerns beyond clinical trials. Raising a question, weighing evidence, recording a verdict — that pattern is generic.

Lalam: They're drawing on forensic modeling through TIDO, so every assessment is a case. Evidence supports or challenges hypotheses, evaluation is an activity, and all of it connects back to PROV-O semantics.

Lu: The ontology distinguishes different kinds of signals — retraction notices, corrections, expressions of concern, peer comments. Each is its own class, so the graph can record precisely what kind of post-publication attention a paper received.

Jane: And the agents are separated too. Human reviewers and automated agents are both recorded as the sources of the hypotheses they generate.

Tom: Without that separation, you couldn't compare machine decisions with human ones. Both sets of hypotheses live in the same structure, which makes the comparison direct.

Jane: Exactly. That parallel design is what powers the disagreement analysis later.

Tom: The model also gives study design evidence and registry evidence their own properties — registration identifiers, dates, recruitment timelines. That tells you the ontology was shaped by what the tool actually had to extract.

Meng: They validated the ontology with a standard pitfall scanner and turned their competency questions into SPARQL queries. Verification like that gives confidence the model behaves as claimed.

Lu: One more thing I'd flag: the authors argue several of their checks are universally applicable to all academic literature, not just randomized trials. Retractions and expressions of concern exist in every field.

Jane: Once you have a structure like that, the natural step is to fill it with real assessments and publish them as a knowledge graph. And that's exactly what RIPE-KG does.

Page 6 of the Paper (Discussing Page 11): Tom: The knowledge graph is the public face of the entire project. At the time of writing it contains 140 assessments of 95 distinct publications, and more than 1,200 individual integrity hypotheses.

Jane: Reviewer anonymity is handled with pseudonymised IDs and roles. The graph can distinguish different reviewers without exposing them, which is sensible given how contentious integrity assessments can be.

Meng: The author disambiguation problem is substantial, and the paper is transparent about the numbers. GROBID extracted over 1,200 author mentions from the PDFs, which collapsed into 875 distinct author instances.

Lu: Of those, 766 got linked to SemOpenAlex author identifiers using owl:sameAs links. That's what enables federated queries across their graph and the wider scholarly graph.

Tom: So roughly 12 percent stayed unlinked.

Lu: Yes, 109 authors, and they don't hide it. Missing OpenAlex metadata and mismatched author lists are the reasons given.

Tom: They also note that the OpenAlex API proved more reliable than the SemOpenAlex SPARQL endpoint, a practical detail that will save people headaches.

Jane: The transformation pipeline is declarative. JSON logs from the tool map into RDF through YARRRML rules, so the graph is regenerated from the process records rather than hand-assembled.

Lalam: For me, the web interface matters as much as the SPARQL endpoint. A knowledge graph only experts can query is a walled garden. The browser view lets editors and reviewers see the assessments and the evidence behind them.

Meng: And because the graph is aligned with SemOpenAlex from the start, you can combine integrity outcomes with fields of study, citation data, and author networks. That's where the analytical power comes from.

Jane: Which brings us to the analysis section. The numbers there are genuinely provocative, because they quantify just how much disagreement exists at every level of the process.

Page 7 of the Paper (Discussing Page 13): Tom: The analysis section is the payoff. Across 514 paired automated and human outcomes, they agreed 86 point 4 percent of the time and disagreed 13 point 6 percent of the time.

Jane: The distribution of those disagreements is the real story. Retraction checks were close, at 8 point 5 percent disagreement. Research-team-related checks ran around 15 percent. And study registration hit 26 point 2 percent disagreement — the weakest spot by far.

Lu: And that makes clinical sense. Judging whether registration was properly timed requires knowing how trials actually operate — whether a delay is acceptable, whether the reported timeline is plausible. That's expertise an LLM can't reliably pull from a PDF.

Meng: The inter-reviewer numbers are even more striking. Across the 22 publications with multiple assessments, reviewers agreed with each other 94 point 4 percent of the time on research-team concerns, but only 44 point 4 percent of the time on registration.

Jane: More than half of the time they disagreed.

Meng: More than half. On a question that can determine whether a trial gets included in an evidence synthesis.

Tom: It's also interesting that the paper suspects some reviewers accepted the automated retraction suggestions without doing the extra check of publisher websites, which the tool can't always reach. The human side can be too trusting too.

Jane: The overall verdicts tell a similar story. Of the 22 works reviewed more than once, over a quarter drew different overall conclusions from different reviewers.

Lalam: My favorite detail is the single publication that received eight assessments spanning all three verdict categories — no concerns, some concerns, serious concerns. That example is the entire argument for provenance in miniature.

Meng: Without an audit trail, that paper looks like noise. With the trail, you can study why experienced people read the same evidence so differently.

Lu: It also pushes back on any idea that the eye should replace the reviewers. The data says the opposite — you need the human, and you need the record of what the human did.

Jane: Then the paper turns to what all this costs to run. And that's the question everyone asks about eye systems in practice.

Page 8 of the Paper (Discussing Page 15): Tom: The cost section is wonderfully concrete. For the entire two-month pilot period, the total operational cost was about.53, and that includes everything.

Jane: The largest line item was infrastructure — a single virtual server running all the services in Docker containers, at. The LLM calls, using Gemini 2 point 0 Flash through Google eye Studio, came to just.34.

Meng: Doing the arithmetic, that's below ten cents per paper analyzed. When the authors say this makes integrity assessment scalable, they show the invoice to prove it.

Lalam: Then the limitations section brings the tone back to earth. Only four of the INSPECT-SR checks are automated so far, and the tool inherits whatever gaps exist in OpenAlex, SemOpenAlex, Retraction Watch, and PubPeer.

Lu: Those external sources are real dependencies. If Retraction Watch is incomplete, or publication data in OpenAlex is wrong, the assessments reflect it. The paper acknowledges this directly.

Jane: There's also the firewall problem. The tool can't always reach publisher websites to verify retraction status, which is why the guidance tells reviewers to do that extra check manually.

Tom: The honest framing is that this is decision support with a recorded audit trail, not an automated judge of scientific trustworthiness. Given the disagreement data we just discussed, that's the right line to draw.

Meng: And the federated query examples on this page show the payoff of linking everything together. They pull endocrinology publications and their integrity outcomes straight out of the combined graph.

Lu: So a funder or a journal could ask, across an entire field, which publications carry integrity concerns. That query is now routine.

Jane: The authors close by drawing conclusions that pull the threads together rather than overclaiming. And that's where we should land too.

Conclusion: Tom: We've walked through the whole pipeline, so let's pull the threads together. What this paper delivers is an end-to-end system for research integrity assessment: an eye-assisted tool, an ontology for provenance, and a public knowledge graph holding 140 assessments of 95 trial publications.

Jane: The core lesson is that integrity assessment is variable by nature, and the right response to that variability is to record it openly rather than smooth it over. Disagreement becomes data.

Lu: For clinical evidence, that means a realistic path toward scaling integrity checks that currently take experts hours per paper, at a cost that's remarkably low.

Meng: For the knowledge graph community, it's a demonstration that trustworthiness can be modeled using established standards — PROV-O, TIDO, the SPAR ontologies — rather than building in isolation.

Lalam: And it's a template for any field that depends on published evidence, from education research to environmental policy. The plumbing is generic even though the checks are clinical.

Tom: The human-in-the-loop result is the thing I'll carry with me. The system found real value, but the disagreements with humans, and among humans, make the case for keeping people central to the process.

Jane: The permanent identifiers, the documented ontology, the queryable graph — this is infrastructure built to be extended. The authors are explicit in inviting other integrity tools to plug into the same framework.

Milan Markovic, Goutham Indukuri, Somayajulu Sripada, Colby J. Vorland, Jack Wilkinson, Clare Robertson, Mark Bolland, Andrew Grey, Miriam Brazzelli, Alison Avenell

University of Aberdeen · Indiana University · University of Manchester · University of Auckland

cs.AI

Submitted: 2026-08-07

Updated: 2026-08-10

Comments: 16 pages

Code: https://github.com/perma-id/w3id.org

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 70/100

The gist: This paper, authored by Milan Markovic, Goutham Indukuri, Somayajulu Sripada, Colby J.

Key concepts

Provenance knowledge graph: A machine-readable record of how each integrity assessment was conducted, including evidence, automated suggestions, and human decisions. It allows tracing why a trial was flagged, ensuring transparency and enabling comparison between human and AI judgments.
INSPECT-SR checklist: A Cochrane-endorsed framework for assessing research integrity in systematic reviews. It guides reviewers through questions about trial registration, data trustworthiness, and other integrity concerns, and is used by the INSPECT-AI tool to structure assessments.
Human-in-the-loop: A design where an AI tool proposes assessments but a human must confirm or override each decision. This ensures final judgments are made by people, addressing disagreements between AI and humans and maintaining accountability in the integrity assessment process.

Terminology

Provenance — 64× in this paper · explained in 4 episodes
Knowledge Graph — 35× in this paper · explained in 7 episodes
Ontology — 35× in this paper · explained in 2 episodes
Trace — 21× in this paper · explained in 2 episodes
Knowledge graphs — 16× in this paper · explained in 2 episodes
Research Integrity Provenance and Evidence — 12× in this paper
Accountability — 6× in this paper · explained in 2 episodes
Core — 6× in this paper · explained in 2 episodes
Aggregation — 5× in this paper · explained in 1 episode
Economic and Social Research Council — 2× in this paper
Human-in-the-Loop — 2× in this paper · explained in 3 episodes
Threat Intelligence Decision Ontology — 2× in this paper
Data Provenance — 1× in this paper · explained in 2 episodes
Generative AI — 1× in this paper · explained in 2 episodes
Graph Construction — 1× in this paper · explained in 3 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
National Institute for Health and Care Research — 1× in this paper
Paper Review — 1× in this paper · explained in 3 episodes
Scottish Intercollegiate Guidelines Network — 1× in this paper
WHO International Clinical Trials Registry Platform — 1× in this paper

Summary

This paper, authored by Milan Markovic, Goutham Indukuri, Somayajulu Sripada, Colby J. Vorland, Jack Wilkinson, Clare Robertson, Mark Bolland, Andrew Grey, Miriam Brazzelli, and Alison Avenell, presents an end-to-end pipeline for authoring and managing transparent research integrity assessments of randomised clinical trial (RCT) publications using LLM-assisted tools and provenance knowledge graphs.

The abstract states: "Systematic reviews of Randomised Controlled Trials (RCTs) are routinely used as evidence for clinical care guidelines. Such evidence has to meet high research integrity standards to prevent low quality or false research outputs influencing the clinical care. However, assessing research integrity of published RCTs is a complex process requiring manual effort, and potentially resulting in diverse opinions of the human assessors. This paper describes INSPECT-AI, an LLM-based interactive tool that assists human reviewers with research integrity assessments of published RCTs based on the community approved INSPECT-SR framework, and the Research Integrity Provenance and Evidence ontology (RIPE-O) for documenting the provenance of the assessment process. In addition, we present the Research Integrity Provenance and Evidence knowledge graph (RIPE-KG), an initial set of 140 expert research integrity assessments of 95 RCT publications generated by INSPECT-AI and described using RIPE-O."

The paper is situated in the field of metascience, which involves the study of scientific practice and addresses challenges related to reproducibility and academic integrity. The authors note that the UK Research Integrity Office sets out five principles of research integrity including honesty, rigor, transparency and communication, care and respect, and accountability.

The motivation is grounded in concrete impacts: "Prior work by members of our research team demonstrated how 27 RCTs with integrity concerns were incorporated into 88 systematic reviews or clinical guidelines, changing findings in over half - 87% with a substantial change in the direction of effect. Additionally, Cochrane reviewers applying an integrity assessment checklist removed 25% of trials, requiring one third of systematic reviews' findings to be updated."

The problem is that "manual completion of checklists and statistical tools is time-consuming, which prevents the scalability of their application. For example, reviewers have to manually search for retraction notices or any relevant comments, assess the co-authors profiles by inspecting their previous publications, and compare information from the publication with other external information."

The authors argue that "research integrity assessments are inherently susceptible to variability, as evaluators following the same guidelines may employ subjective judgements. Therefore, we argue that FAIR principles should apply to the publication of provenance traces of such assessments, providing the standardised infrastructure needed to make their results transparent, reproducible, and reusable across the scientific community."

The paper presents three main components of the RIPE Observatory:

INSPECT-AI: an AI tool enabling researchers to perform research integrity assessments and to generate semantic provenance traces documenting the assessment processes and their outcomes.
RIPE-O: Research Integrity Provenance and Evidence Ontology (RIPE-O) for documenting the provenance of the integrity assessment process.
RIPE-KG: Research Integrity Provenance and Evidence Knowledge Graph (RIPE-KG) with the web-based GUI, containing at the time of writing provenance traces describing 140 integrity assessments of 95 publications.

The paper notes that while the core semantic contributions of this paper are RIPE-O and RIPE-KG, the INSPECT-AI assessment tool is included because it informed the development of RIPE-O and completes the proposed pipeline. The authors also envision RIPE-O as an ontology for integrating outputs from other existing and future research integrity assessment tools into RIPE-KG or similar knowledge graphs.

The paper reviews existing scholarly knowledge graphs, noting that Knowledge graph technologies have been successfully used to model, integrate, and manage large volumes of scholarly data from heterogeneous sources. Examples include OpenAIRE Research Graph, Scholia, Semantic Scholar, Open Research Knowledge Graph, and SemOpenAlex. However, to the best of our knowledge, none of the existing scholarly knowledge graphs integrate data related to publication trustworthiness or other research integrity assessments.

The authors observe that To date, discovering problematic RCTs remains a highly manual effort. Research integrity 'sleuths' may alert journals to untrustworthy publications, but "publishers can be very slow to retract articles or post expressions of concern on their webpages, if they do at all. A high proportion of untrustworthy publications remain unflagged by journals and publishers years after notification."

Existing assessment tools include "TRACT, RIA, CPC-TST, and REAPPRAISED checklists, alongside early experimentation with semi-automating the TRACT checklist using GPT-4o. However, these mostly manual tools are not endorsed by Cochrane and do not produce semantic data. The INSPECT-SR checklist, by contrast, was developed by international integrity researchers through a consensus and prototype testing process. The 21 checks cover four domains: inspecting post-publication notices; inspecting reported conduct, governance and transparency; inspecting text and figures (for image manipulation and text duplication); and inspecting results, which mainly relate to numerical data and statistical results. This checklist is endorsed by the Cochrane Collaboration."

The work resulted from interdisciplinary efforts of systematic reviewers, clinicians, statisticians, and computer scientists. The authors adopted an agile software development methodology and produced a number of early iterative software prototypes and demos. A key finding was that real, semi-functional prototypes resembling the final product were more effective than paper prototypes or detailed technical explanations.

The human-in-the-loop design was influenced by enhanced understanding of AI limitations among non-technical team members. The biomedical researchers provided a non-trivial set of 50 diverse examples of existing real-world problematic publications and detailed guidance on the assessment process.

For RIPE-O development, the authors followed the four-stage Linked Open Terms methodology (LOT) and formed competency questions guided by the selected INSPECT-SR checks and framed by the 7W's of Provenance (Who, What, Where, Which, Why, When, How). The ontology extends "the Threat Intelligence Decision Ontology (TIDO), which adopts a forensic analysis approach to documenting provenance of decision-making processes and aligns with the PROV-O ontology for documenting provenance traces. SPAR ontologies including FABIO and CITO were reused to model additional bibliographic information."

The INSPECT-AI workflow consists of five steps: "1. Publication Upload; 2. Automated Evidence Aggregation; 3. Automated Evaluation; 4. Human Review; 5. Provenance Store & User Interface."

In the evidence aggregation step, "INSPECT-AI extracts structured metadata using GROBID, including the publication DOI, title, authors, and reference list. In parallel, INSPECT-AI uses LLM (GEMINI 2.0 Flash) to extract additional information such as registration identifiers and the study timeline (i.e., start and end of the recruitment, trial dates) from the publication's text. Each LLM-generated piece of evidence is accompanied by a short explanation of why particular values were extracted so that reviewers can later quickly identify and verify the extracted information within the publication."

External evidence sources include the Retraction Watch Database, and PubPeer for external evidence linked to the publication and, for trial registrations, the ClinicalTrials.gov API and the WHO ICTRP Search Portal.

For the automated evaluation, "each evidence piece is evaluated against the relevant INSPECT-SR check using conditional logic with pre-defined rules, and the system generates suggested outcomes stating whether concerns have been found, or whether the evidence is unclear. The INSPECT-AI web interface presents each INSPECT-SR assessment check together with the collected evidence and suggested outcomes. Human reviewers accept, modify, or override these suggestions before submitting their final assessment. All provenance information is logged in a PostgreSQL database."

The tool was "piloted with 13 volunteers recruited from the Institute of Applied Health Sciences, University of Aberdeen, Centre for Biostatistics, University of Manchester, and the Indiana University School of Public Health-Bloomington during October to November 2025. Participants used INSPECT-AI independently to assess 69 publications and generated 104 assessment traces."

Key pilot statistics: "Most participants (77%) had experience of conducting systematic reviews, with 31% of participants reporting that they conduct reviews frequently (n=2) or very frequently (n=2), and (38.5%) had ten or more years' experience of conducting reviews. Only 61.5% of participants had previously undertaken integrity assessments of RCTs. Usability results: Just under half of the participants (46%) found the tool easy (n=3) or very easy to use (n=3) and 69% found the tool either fairly user-friendly (n=6) or very user-friendly (n=3). None of the participants reported that they found the tool difficult to use or that the tool was not user-friendly. The response time of the tool was rated as quick (n=5) or very quick (n=4) by 69% of participants."

RIPE-O's aim is "to document provenance of the research integrity assessment process combining automated and human-generated outputs. Although the ontology design has been driven by the INSPECT-AI use case, the modelled concepts remain generic. RIPE-O does not prescribe which questions are investigated."

Core modelling concepts include: ripe:Work for the assessed publication; ripe:ResearchIntegrityAssessment as a case; ripe:IntegrityAssessmentQuestion for investigated questions; ripe:IntegrityAssessmentHypothesis for results of evaluation activities; ripe:IntegrityAssessmentEvidence for evidence items; ripe:HumanReviewer and ripe:AutomatedAgent for agents. Provenance traces are documented using activities, entities and agents following PROV-O patterns.

Evidence types modelled include "ripe:CorrectionNotice, ripe:RetractionNotice, or ripe:ExpressionOfConcern but also evidence generated from the publication's content or external resources. This includes trial design details (ripe:StudyDesignEvidence) such as start and end dates of participant recruitment mentioned in the article, and trial registration data (ripe:RegistryEvidence) extracted from the trial registries. The ontology also models ripe:PeerComment."

Validation was performed using the OntOlogy Pitfall Scanner (OOPS!) to detect potential modelling errors and validated the ontology by converting the competency questions into SPARQL queries.

RIPE-KG contains provenance traces of research integrity assessments performed using INSPECT-AI during tool evaluation, as well as assessments performed by our expert team members (research sleuths). For reviewer anonymity, each reviewer is identified by a pseudonymised reviewer ID and their role. Content includes "140 assessments of 95 distinct publications, 927 distinct author entities representing the authors of assessed publications, references and PubPeer comments, and 1,221 research integrity hypotheses (INSPECT-SR check outcomes) generated by both the tool and human reviewers."

Author disambiguation is handled by using the OpenAlex API: "GROBID extracted 1,202 publication-author mentions from the assessed publications, which were disambiguated into 875 distinct RIPE author instances. Of these, 766 are linked to SemOpenAlex, while 109 (12.5%) remain unlinked due to missing OpenAlex author metadata."

INSPECT-AI implements four INSPECT-SR checks: associated retraction; expression of concern or post-publication notice; concerns from other studies by the research team; and concerns about timing/absence of study registration.

The analysis found: "We have analysed 514 assessments × question pairs for which both automated and human-reviewed outcomes are available. Of these, 444 (86.4%) show agreement between automated and human-reviewed outcomes, while 70 (13.6%) show disagreement. The disagreement pattern is uneven across questions: retraction status shows 11 disagreements out of 129 pairs (8.5%), post-publication notices 5 out of 127 (3.9%), research-team-related concerns 20 out of 128 (15.6%), and study registration 34 out of 130 (26.2%)."

Human-human agreement was also analysed: "We have analysed 22 publications where more than one assessment was recorded... reviewers agree in 15 of 18 cases on retraction status (83.3%), 16 of 19 on post-publication notices (84.2%), 17 of 18 on research-team-related concerns (94.4%), but only 8 of 18 on study registration concerns (44.4%)."

The authors interpret these results: "These results suggest that study registration is the least agreeable question because, unlike retractions or post-publication notices, it often requires expert domain knowledge to determine whether the reported registration, recruitment, and study timelines are compatible with acceptable clinical trial practice. Disagreements on the retraction status are also interesting, because this may indicate that some human reviewers sided with the automated INSPECT-AI suggestions without following the additional guidance to check publishers' websites. This extra step is suggested due to the current INSPECT-AI inability to bypass the firewall protections of various publishers' websites and, hence, can retrieve the retraction status only from the PDF of the publication or the Retraction Watch Database, which may be incomplete."

For overall assessments: "RIPE-KG contains 140 human-reviewed overall assessments for 95 works. Of these, 22 works were reviewed more than once. Among those 22 repeatedly reviewed works, 16 (72.7%) received the same overall human assessment across repeated reviews, while 6 (27.3%) showed disagreement between reviewers. The clearest disagreement example is DOI 10.1111/bjdp.12503, which appears in 8 human-reviewed assessments spanning all three overall assessment categories: no-concerns, some-concerns, and serious-concerns."

The conclusion from this analysis: Overall, the results support the need for provenance tracking in the research integrity assessment context due to disagreements not only among human assessors but also between human and automated assessments.

RIPE-KG supports basic author-network analysis including identifying authors who frequently co-author publications with integrity concerns via SPARQL queries. The KG also supports federated queries with SemOpenAlex to answer questions such as What are the research integrity assessment results of publications from a specific field of study?

During the pilot study period, "the total operational cost of INSPECT-AI was approximately US 79.53. The largest component was infrastructure: all services ran as Docker containers on a single VPS (8 vCPU, 32 GB RAM), costing 77 for the period. LLM costs were modest with Gemini 2.0 Flash, accessed through Google AI Studio, handled structured data extraction from PDFs at a cost of 1.34. At this level of expenditure, the marginal cost per paper analysed was below 0.10."

The authors identify three main limitations: "The current version of the INSPECT-AI prototype does not provide AI support for all INSPECT-SR checks. In addition, because of our reliance on OpenAlex for author disambiguation, RIPE-KG will inherit any insufficiencies present in this external database. The functionality of the tool is also affected by the quality of external information sources such as SemOpenAlex knowledge graph, Retraction Watch Database, and PubPeer."

The paper introduces "the first end-to-end pipeline for generating provenance traces of research integrity assessments and publishing them as open knowledge graphs in accordance with FAIR principles. Our approach advances the wider agenda of scholarly knowledge graphs and provides reusable technologies for integrating outputs from existing and future research integrity assessment tools into provenance knowledge graphs that can support new research integrity metrics and research-output appraisal by funders, publishers, and other end users."

The findings "highlight the inherent complexity and variability of research integrity assessment, with discrepancies between human and automated assessments. These results support the need to incorporate human validation into LLM-based decision support systems to ensure the reliability and accountability of their outputs. The authors note that Given the limited availability of human experts in this context, the proposed approach is particularly relevant, as it enables the preservation and sharing of validated assessments. As the KG grows over time through the addition of tools covering more disciplines, we envision this resource becoming a key enabler of new semantic approaches for transparent research integrity assessments."

Future plans include "extending the coverage of INSPECT-SR checks within the INSPECT-AI tool, as well as exploring the application of RIPE-O for documenting research integrity assessments in a broader biomedical context and across other disciplines. The authors will continue to engage with systematic reviewers and research sleuths through additional community-based deployments of INSPECT-AI to expand the RIPE-KG knowledge graph. They will also investigate advanced graph-based selection mechanisms that leverage RIPE-KG data to automatically discover potentially untrustworthy publications for future human-based assessments."

The work was supported by the Economic and Social Research Council (ESRC) [grant number UKRI1083] and by alumni and friends in establishing the University of Aberdeen's Interdisciplinary Institute. The paper declares that "GenAI tools have been used to generate LaTeX formatting code for tables, figures and listings and for text shortening suggestions. GenAI tools have also been used to support programming activities including query and code generation during the development of INSPECT-AI and RIPE-KG web interface."

Improvements for AI systems

Add explicit provenance tracking to every AI decision: The AI will record each evidence item, the reasoning chain, the LLM prompt/version, and the human reviewer’s accept/modify/override action. This makes every integrity assessment auditable and reproducible in a knowledge graph.
Separate automated suggestions from human verdicts: The AI will expose both outcomes and will explicitly quantify agreement/disagreement rates per INSPECT-SR check, so users can see where automation is reliable (e.g., retraction detection) and where it needs escalation (e.g., study registration).
Implement a disagreement-aware learning loop: When the human overrides an automated suggestion, the AI will store that override as a provenance trace, cluster similar disagreement patterns, and use them to retrain or adjust extraction rules and conditional logic.
Decompose complex checks into sub-questions: For a check like “study registration concerns” that showed 44.4% human-human agreement, the AI will break it into specific sub-evidence: registration number present, registry name, recruitment start/end dates, ethics statement, and timeline consistency. It will then show each sub-question separately, reducing ambiguous holistic judgments.
Per-question confidence calibration: The AI will output a confidence score for each suggested INSPECT-SR outcome, calibrated using known agreement rates from RIPE-KG. For low-confidence checks, it will automatically require human review; for high-confidence checks, it will still present evidence but faster.
Fetch and compare publisher metadata beyond the PDF: Because retraction status disagreements may come from missing publisher website data, the AI will add a retrieval module that queries Crossref, journal websites, and Retraction Watch, and will mark whether publisher confirmation was obtained or unavailable.
Use RIPE-KG as a memory for future assessments: The AI will query the knowledge graph for prior assessments of the same authors, institutions, affiliated trials, or duplicate/related publications. It will then surface relevant prior concerns—e.g., “a co-author has another RCT with serious integrity concerns”—without making a final judgment automatically.
Generate human-readable provenance summaries: The AI will produce a natural-language report per assessment that explains which evidence supported each outcome, which evidence was unclear, where the human changed the AI suggestion, and which external databases were consulted. This report is directly citable and reusable by systematic reviewers.
Enable federated queries over assessment data: The AI will support SPARQL queries that join RIPE-KG with SemOpenAlex, so users can ask questions like “Show integrity assessments for publications in this field, grouped by author network” or “Find all RCTs by these co-authors with unresolved retraction concerns.”
Active-learning selection of publications for review: Using AI graph traversal over RIPE-KG, the system will identify clusters of publications that are likely untrustworthy (shared authors, similar text/image anomalies, missing registration) and recommend those to human reviewers, prioritizing scarce expert attention.
Multi-agent verification of extracted facts: The AI will use separate LLM passes for registration numbers, study timelines, author lists, and retraction notices, then cross-check conflicting extractions against the source PDF and external APIs. Only agreed-upon facts will be attached as evidence; conflicts will be flagged for human resolution.
Track reviewer demographics and calibrate group reliability: The AI will pseudonymously record reviewer experience, specialty, and completion time. It will use this data to compare individual judgments against group consensus in RIPE-KG and warn when a reviewer’s assessment pattern deviates significantly for low-agreement checks.
Extend evidence types to images and text duplication: The AI will add modules for image manipulation detection and near-duplicate text detection across publications, converting those checks into provenance-typed evidence (e.g., ripe:ImageEvidence, ripe:TextDuplicationEvidence) in RIPE-O.
Provide a cost-controlled human-in-the-loop interface: The AI will show live per-publication LLM and infrastructure costs, automatically fall back to cheaper models for simple extraction tasks, and reserve expensive models for low-confidence or override-rich checks, maintaining the sub-0.10 per-paper target.
Publish assessments as FAIR knowledge graph resources: The improved AI system will automatically mint persistent identifiers for each assessment trace, expose the provenance graph via a web GUI, and allow other systems to consume those traces, enabling cross-institution integrity monitoring.

Abstract

Systematic reviews of Randomised Controlled Trials (RCTs) are routinely used as evidence for clinical care guidelines. Such evidence has to meet high research integrity standards to prevent low quality or false research outputs influencing the clinical care. However, assessing research integrity of published RCTs is a complex process requiring manual effort, and potentially resulting in diverse opinions of the human assessors. This paper describes INSPECT-AI, an LLM-based interactive tool that assists human reviewers with research integrity assessments of published RCTs based on the community approved INSPECT-SR framework, and the Research Integrity Provenance and Evidence ontology (RIPE-O) for documenting the provenance of the assessment process. In addition, we present the Research Integrity Provenance and Evidence knowledge graph (RIPE-KG), an initial set of 140 expert research integrity assessments of 95 RCT publications generated by INSPECT-AI and described using RIPE-O.

Sources

The Semantic Scholar Open Data Platform