MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis".

Jane: The gist MeddeX presents a system that combines a QLoRA fine-tuned LLaMA-3 (8B) model with Retrieval-Augmented Generation (RAG) and Gemini Vision for evidence-backed medical report analysis.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper titled "MeddeX: A Retrieval-Augmented LLaMA-three System with Gemini Vision for Evidence-Backed Medical Report Analysis <ref:2610.20632#pg1,MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence>." It looks like they've put a QLoRA fine-tuned LLaMA-three model together with Retrieval Augmented Generation and Gemini Vision to analyze medical reports <ref:2610.20632#pg1,a QLoRA fine-tuned LLaMA-3>.

Jane: That sounds pretty complex. So what's the main goal here, Tom? What problem are they trying to solve with this system?

Tom: The whole idea is tackling the slow, error-prone, and hard-to-scale manual review of modern medical reports. They combine a fine-tuned LLaMA-three model with RAG and Gemini Vision to give accurate summaries that actually stick to evidence in the report <ref:2610.20632#pg1>.

Lu: It's interesting how they are moving beyond just using a standard LLM for summarization without any source grounding, which is something they say is limiting clinician trust <ref:2610.20632#pg2>.

Meng: So, what does this actually mean for a doctor sitting at their desk? Is this going to be something they can just plug into their existing workflow right now?

Tom: Well, the system handles different kinds of inputs—text PDFs, scanned image-based ones, and even direct radiological images. It uses a pipeline that checks for corrupted text first before sending it to the vision layer if needed <ref:2610.20632#pg1>.

Jane: And what about the authors? It lists Rajdeep Pandhare, Rohit Kundaikar, Yakub Kojar, and Shreyansh Tiwari from the Department of Artificial Intelligence and Data Science at Rajiv Gandhi Institute of Technology.

Lu: They're coming from a place where they are working on model training and representation learning <ref:2610.20632#pg3>. Their focus seems to be on how to make these grounded outputs reliable for clinical use.

Meng: Reliability is key, but running a system like this at scale sounds resource intensive. How big is this thing, and what's the immediate deployment plan?

Tom: They say they reduced VRAM requirements for fine-tuning by approximately four times compared to full BF16 finetuning <ref:2610.20632#pg3>, which helps with the cost of training. And they use QLoRA, which is a technique for making large models more efficient.

Jane: So the authors are showing that you can get strong performance by combining these different AI components rather than just relying on one thing.

Lu: Yes, it’s about showing that integrating different modalities and techniques leads to better results when you need verifiable answers <ref:2610.20632#pg2>.

The paper's summary: Tom: Okay, so the core of MeddeX is this hybrid extraction pipeline. It starts with ingestion, then garbage detection to spot bad text, then smart extraction routing pages to either standard PDF tools or Gemini Vision OCR if they look corrupted <ref:2610.20632#pg1>.

Jane: That garbage detection sounds really clever for handling messy medical documents. Can you explain what triggers that check?

Tom: It checks for things like UTF-sixteen hex artifacts or high ratios of non-printable characters in the text, and if it sees those, it reroutes that page to Gemini Vision OCR instead of just trying to read the corrupted text <ref:2610.20632#pg1>.

Meng: If you're running this in a production environment, what happens when the garbage detection fails or flags a lot of pages as bad? Does it just fail, or does it have a fallback?

Tom: It reroutes those corrupted pages to Gemini Vision OCR for extraction, which is why they call it smart multimodal extraction <ref:2610.20632#pg1>. So the system tries multiple paths until it gets clean text.

Jane: And once they get the text extracted, how do they actually use that information to give an answer? That's where the RAG comes in, right?

Tom: Exactly. They chunk that extracted text into small pieces, like five hundred characters long with a fifty character overlap to keep local context <ref:2610.20632#pg3>. Then they embed those chunks using all-MiniLM-L6-v2 and store those embeddings in Supabase <ref:2610.20632#pg3>.

Lu: Storing three hundred eighty-four dimensional vectors is a common approach for dense retrieval, but they are using that to enable report scoped retrieval later, which I think is the most important part for clinical accuracy <ref:2610.20632#pg3>.

Jane: So when a user asks a question about a specific report, the system doesn't search everything; it narrows down the chunks to only those related to that active report <ref:2610.20632#pg3>. That seems very smart for keeping things focused.

Tom: It does. Then they retrieve the top-k most similar chunks using an RPC called match chunks by report, and then they feed those retrieved passages along with the chat history into the fine-tuned LLaMA-three model for generation <ref:2610.20632#pg3>.

The paper's improvements: Tom: They implemented a few specific technical mechanisms that make this system robust. First, they have this garbage-text detection function, is garbage text, which guards against silently corrupted text extractions <ref:2610.20632#pg1>.

Jane: That sounds like a safety net for the extraction process. What else are they focusing on improving in the actual generation part? Because that's where we need to know if it actually gives us good answers.

Tom: They use a QLoRA fine-tuned LLaMA-three model, specifically the checkpoint LLaMA3-Med42-8B, which they say shows consistent improvements over their base model across several metrics <ref:2610.20632#pg4>.

Lu: The results they show are pretty solid. They report gains across ROUGE-one ROUGE-two and ROUGE-L scores, getting up to +eleven point eight percent in the BLEU score <ref:2610.20632#pg4>. That indicates the fine-tuning is making it much better at generating precise phrases compared to just using the base LLaMA3-Med42-8B <ref:2610.20632#pg4>.

Meng: If you're looking at deployment, what about speed? When a doctor asks a question, how fast is this whole process from upload to answer? We need low latency for clinical tools.

Tom: They use nucleus sampling with top-p set to zero point nine and temperature set to zero point three for Q andA, which keeps the output focused and somewhat consistent <ref:2610.20632#pg4>. And they keep the maximum number of new tokens for Q andA at three hundred with a larger limit of six hundred for report comparisons <ref:2610.20632#pg4>.

Jane: That’s helpful detail on the generation side. It sounds like they are balancing being creative enough to generate a good answer with being constrained enough to stay grounded in the retrieved chunks.

Lu: They also explicitly mention that they avoid hallucination by conditioning the LLaMA-three generator on those retrieved passages, which is what their RAG objective defines as Eq <ref:2610.20632#pg1>. one <ref:2610.20632#pg2>.

Meng: So, the main improvement here seems to be making sure that when it does generate something, it has a traceable source for every claim. That's a big deal for clinical adoption if it’s not just guessing at facts.

Conclusion: Tom: To wrap up MeddeX, this paper presents a retrieval-augmented LLaMA-three system with Gemini Vision that handles various medical document types and aims to provide evidence-backed summaries <ref:2610.20632#pg1,a retrieval-augmented LLaMA-3 system with Gemini Vision>. The main contribution is the integrated pipeline combining garbage detection, smart extraction, chunking with embeddings in Supabase, and report-scoped retrieval <ref:2610.20632#pg3>.

Jane: It’s a system that’s designed for clinical utility by ensuring the generated answers can be traced back to specific source chunks in the original reports <ref:2610.20632#pg3>. This moves us closer to having tools that clinicians can actually trust with their patients' information.

Lu: The work shows how combining domain pretraining, like BioBERT two or Longformer four, with a RAG framework allows for outputs to be traced to source evidence <ref:2610.20632#pg2>. It’s about grounding the language model in the actual documents.

Meng: From an engineering standpoint, what's the main constraint they acknowledge? They mention that for the comparison endpoint, they currently truncate each report to one thousand five hundred characters of context for the prompt <ref:2610.20632#pg4>. That’s a limitation we have to keep in mind when deploying it.

Tom: Right, so it handles a lot—text and images—and shows strong gains over the base model, especially in BLEU score improvements of seventy-one point four percent <ref:2610.20632#pg4>. It’s a production-oriented system built with efficiency in mind through QLoRA and VRAM reductions <ref:2610.20632#pg3>.

Jane: So, the big picture is that they are building a comprehensive system for analysis, not just a single piece of software, by focusing on how the retrieval mechanism connects to the generation step <ref:2610.20632#pg4>.

Lu: They also touched on future work involving model distillation to support inference on edge servers or low-VRAM GPUs for resource-constrained settings <ref:2610.20632#pg4>. That shows they are thinking about how to make this accessible beyond big data centers.

Meng: And I'm looking forward to seeing if they can do that real-time summarization inside actual electronic health record systems, which would really change the workflow for doctors one <ref:2610.20632#pg2>.

Tom: Exactly. So the MeddeX paper is a solid example of how we can take established models and build complex, evidence-backed tools by adding smart retrieval and multimodal capabilities to solve real problems in healthcare <ref:2610.20632#pg1>.