Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning".

Jane: Here's the gist of the paper. The gist: Using an automatically curated dataset can substantially increase the performance of deep learning algorithms, and it is suggested to use all the data rather than only using the accurate subset.

Tom: Ultrasound (US) remains a standard of care for imaging evaluation of thyroid nodules Artificial intelligence (AI), and particularly deep learning (DL) have been increasingly applied to thyroid US to improve diagnostic accuracy and efficiency Several deep learning applications have been shown to accurately characterize nodules as benign or malignant, and some have even performed better than radiologists When designing deep learning tools for medical image analysis such as thyroid US, large datasets remain key for training a highperforming algorithm However, curating and annotating large datasets of radiologic images is time-consuming and labor-intensive One potential solution to this problem is to use tools and pipelines that automatically annotate images to generate larger medical datasets In our prior work, we created an automated labeling procedure (called MADLaP) that used a combination of tools to annotate images from thyroid ultrasounds The tool harnessed imaging deep learning, natural language processing, and optical character recognition to extract data from pathology reports, radiology reports, and US images Study Design and Datasets The primary hypothesis of the study is that convolutional neural network-based classifiers for thyroid nodules in ultrasound, trained using large, automatically curated datasets perform better than the same classifiers trained using smaller, manually curated datasets To test this hypothesis, three training datasets were created from the development cohort (Manual Set, MADLaP Set, and S1 Set) following the workflow shown in Figure 1 The first dataset (referred to as the ’Manual Set’) was based on the random selection of 378 patients from the study population These data were manually reviewed by a fellowshiptrained radiologist (4 years post-fellowship experience) The second dataset (referred to as ’MADLaP Set’) was constructed using the automated tool from our prior work: we employed a Multistep Automated Data Labeling Procedure (MADLaP) tool for automatic annotation This automatically-curated dataset consisted of 5,228 images (4,970 images for benign nodules and 258 for malignant nodules) The third dataset (referred to as ’S1 Set’) was created using only Stage 1 of the MADLaP pipeline Since Stage 1 operates on more standardized imaging studies, this subset has higher label accuracy but a lower yield than the full MADLaP output Deep Learning Model and Evaluation The deep learning algorithm used was a Faster Regionbased Convolutional Neural Network with ResNet-101 backbone pre-trained on MS COCO dataset The architecture involved six 3 x 3 convolutional filters with ReLU activation functions, five 2 x 2 max pooling layers, a 50% dropout layer for regularization in the training phase, and a fully connected layer with one output and a sigmoid function Focal loss was implemented as the dynamically scaled cross entropy loss to tackle the class imbalance issue in the training dataset Performance was calculated using Area under the receiver operating characteristic curve (AUC), with confidence intervals calculated using bootstrap with 10,000 repetitions Key Findings The deep learning model trained on the manually selected dataset had an AUC of 0.643 (95% confidence interval

CI: : 0.62, 0.66) This was significantly lower than the AUC of the automatically-curated dataset trained deep learning model, 0.694 (95% confidence interval

CI: : 0.67, 0.73, P <.001) The AUC of the accurate subset trained deep learning model is 0.689 (95% confidence interval

CI: : 0.66, 0.72, P <.43), which is insignificantly worse than the AUC of the full automatically-curated dataset The MADLaP-based algorithm achieved an AUC of 0.694 (95% confidence interval

CI: : 0.67, 0.73, P <.001) Furthermore, the MADLaP set model improved from 0.656 to 0.700 when compared to the Manual set model with a batch size of 512 The study concluded that using all the data selected by MADLaP should be done rather than only using the accurate subset Proper utilization of the MADLaP pipeline can significantly improve the performance of classification deep learning algorithms distinguishing malignant thyroid nodules The batch size should also be large enough for the deep learning algorithms to neglect the effect of inaccurate data The experiment proved that using an automatically-curated dataset can substantially increase the performance of deep learning algorithms, and it is suggested to use all the data rather than only using the accurate subset The MADLaP generated dataset, even with 17% noise, is capable of training a better deep learning algorithm than the smaller manually selected dataset A larger batch size positively impacts the automatically-curated dataset more than the manually labeled dataset in our cases This suggests that automated tools like MADLaP are beneficial for developing high-performing deep learning models in this field The best performance of the ’MADLaP set’ is 0.700 (95% confidence interval

CI: : 0.67, 0.73), which is similar and comparable to the benchmark deep learning algorithm's performance on the testing set of 0.70 (95% CI: 0.64-0.75) The study proved the practicality of using automaticallycurated data The MADLaP pipeline does not have meaningful comparisons since this is the only tool that can automatically annotate thyroid datasets with rich information for downstream machine learning tasks This work contributes to understanding how noisy data can be handled in medical image classification tasks The study also examined the impact of batch size on the automatically-curated medical dataset, observing that a larger batch size substantially increased the performance of the deep learning algorithm trained by the noisy automatically-curated dataset This finding aligns with previous studies showing that increasing batch size is a practical way of dealing with noisy data The results demonstrate that MADLaP can significantly improve classification accuracy for thyroid nodules Page 1 --- Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning Jichen Yang1 Jikai Zhang1 Maciej A.

Jane: Mazurowski1,2 Benjamin Wildman-Tobriner2 Department of Electrical & Computer Engineering, Duke University, Durham, NC, USA Department of Radiology, Duke University Medical Center, Durham, NC, USA Page 2 --- The diagnosis of thyroid nodule cancers commonly utilizes ultrasound images.

Tom: Several studies showed that deep learning algorithms designed to classify benign and malignant thyroid nodules could match radiologists’ performance.

Title and authors: Tom: So, we're looking at this paper here titled "Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning," and it’s about using these larger, automatically curated datasets to see if they actually make deep learning models better than the ones trained on smaller, manually labeled sets.

Jane: That sounds like a really practical question because we know getting big datasets for medical imaging is super hard and time-consuming for radiologists to label manually. So, what’s the main point this paper is trying to get across about these automated methods?

Lu: The core idea here is testing the hypothesis that convolutional neural network classifiers trained on large, automatically curated datasets perform better than those trained on smaller, manually curated datasets. It's about seeing if we can use all that data rather than just a small accurate subset.

Meng: So, what does this paper actually test? Are they just comparing accuracy scores or something more? I need to know what the actual numbers look like when you put it into practice.

Tom: They did three training datasets from a development cohort: the 'Manual Set' from random selection, the 'MADLaP Set' built with their automated tool, and then an 'S1 Set' which is just using stage one of that MADLaP pipeline.

Jane: And what’s the biggest difference between those three sets in terms of how they were created? I want to understand how much noise we are dealing with when we move from one set to the other.

Tom: The 'Manual Set' came from randomly selecting three hundred seventy-eight patients, and those images were manually reviewed by a fellowshiptrained radiologist who had four years post-fellowship experience <ref:2602.01020#pg3,were manually reviewed by a fellowshiptrained radiologist>.

Jane: And then the 'MADLaP Set,' that’s the one built using their automated tool—the Multistep Automated Data Labeling Procedure—which they say uses rule-based NLP, optical character recognition, and a deep learning image annotator to grab data from pathology reports and radiology reports.

Lu: The MADLaP Set ended up being five thousand two hundred twenty-eight images total, with four thousand nine hundred seventy of them being for benign nodules and two hundred fifty-eight for malignant nodules. It shows the potential volume of what that automated pipeline can generate.

Meng: That’s a lot of data to process automatically; I wonder how robust that extraction is when you're dealing with all those different report types—pathology, radiology, US images—all at once.

Tom: They also created an 'S1 Set' which only uses stage one of the MADLaP pipeline, and they mentioned that this subset has higher label accuracy but a lower yield than the full MADLaP output.

Title and authors: Jane: So what was the deep learning model used for this comparison, and how did they measure if the automatically curated data actually helped improve things?

Tom: They used a Faster Regionbased Convolutional Neural Network with a ResNet-one hundred one backbone pre-trained on the MS COCO dataset <ref:2602.01020#pg3>. The architecture included six three by three convolutional filters with ReLU activation functions, five two by two max pooling layers, a fifty percent dropout layer for regularization during training, and a fully connected layer with one output using a sigmoid function.

Lu: They also tackled class imbalance in the training data using focal loss as the dynamically scaled cross entropy loss function. That’s important because of how unbalanced medical datasets usually are.

Meng: So, what were the actual performance metrics they used to judge which dataset was superior? We need concrete numbers here, not just qualitative statements about "better."

Tom: They calculated performance using the Area under the receiver operating characteristic curve AUC with confidence intervals calculated using bootstrap with ten thousand repetitions. That’s how they quantified everything.

Jane: Let's look at those results then. How did the model trained on that smaller, manually selected dataset perform compared to the one trained on the automatically curated data?

Tom: The deep learning model trained on the manually selected dataset had an AUC of zero point six four three with a ninety-five percent confidence interval of zero point six two to zero point six six. It was significantly lower than the AUC of the automatically curated dataset model, which got an AUC of zero point six nine four with a ninety-five percent confidence interval of zero point six seven to zero point seven three, and that difference was statistically significant at P less than point zero zero one.

Lu: That means moving from the manually selected data to the automatically curated dataset gave a measurable performance gain in this specific setup, which is what they aimed to prove when they asked whether these datasets are useful for accurate deep learning models.

Jane: But wait, there’s another subset mentioned—the accurate subset trained on only Stage one of MADLaP—and how did that compare <ref:2602.01020#pg1>?

Tom: The AUC for the accurate subset trained deep learning model was zero point six eight nine with a ninety-five percent confidence interval of zero point six six to zero point seven two, but that difference was not statistically significant at P less than point zero four three.

Meng: So, the accurate subset wasn't significantly better than the full automated set when we looked at those specific numbers; that tells us something about how much noise or data volume is actually needed to see a real lift.

Tom: Exactly. The study concluded that using all the data selected by MADLaP should be done rather than only using the accurate subset because it was significantly better than the manually selected set, which improved from an AUC of zero point six five six to zero point seven zero zero when compared to the manual set model with a batch size of five hundred twelve.

Title and authors: Lu: So they’re suggesting that even though there is noise in the automated data—they mentioned that it has seventeen percent noise—the benefit comes from using the whole set and also increasing the batch size to let the algorithm ignore some of that inaccuracy.

Jane: That’s a really important practical implication for anyone trying to build these tools; it suggests that if you can use an automated pipeline like MADLaP, you should not throw away the data just because it's not perfectly clean.

Tom: Right, so the main message is that using all of the data from MADLaP is better than just a highly accurate subset, and making your batch size larger helps handle that noisy data.

Meng: From a deployment standpoint, if we can get an AUC of zero point seven zero zero by training on this noisy dataset with a large batch size, that’s much more valuable than chasing a marginally better score on the smaller set.

Lu: This work contributes to understanding how noisy data can be handled in medical image classification tasks, showing that proper utilization of the MADLaP pipeline can significantly improve classification accuracy for thyroid nodules.

Jane: So, to wrap up this discussion on "Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning," it seems the key message is that leveraging automated tools like MADLaP is beneficial because it provides a much larger dataset, and you should use all of that data rather than just a highly accurate subset.

Tom: Yeah, we showed that even with the seventeen percent noise in the MADLaP set, it's capable of training a better deep learning algorithm than the smaller manually selected dataset <ref:2602.01020#pg2>.

Meng: And remember what they found about batch size—a larger batch size substantially increased the performance of the deep learning algorithm trained by that noisy automatically curated dataset.

Lu: This suggests that automated tools like MADLaP are beneficial for developing high-performing deep learning models in this field, and it shows how to handle noisy data in medical image classification tasks.

Jane: So for anyone listening who doesn't read the paper, the big picture is that automating data curation can substantially increase performance when training deep learning models for thyroid nodules, provided you use all of that generated data.

Tom: That’s it for this one; we’ve talked about how to use automatically curated datasets and what the results show.

Meng: Alright, let’s see what other interesting papers are waiting on arXiv next.

The paper's summary: Tom: So, we’ve been looking at the mechanics of this study—how they compared the manually labeled data against their automated labeling pipeline—and now we need to talk about what all that means for actually using these tools in a real clinic or lab.

Jane: Exactly. The main takeaway here is that just having a big dataset isn't enough, and you can’t just pick the most perfect small part of it to train your AI; you have to use the whole collection generated by the automated system.

Tom: Right. They found that training on that full, noisier automatically curated MADLaP set—even with seventeen percent noise—actually resulted in a better performance score than they got on just the highly accurate subset, which is what we expect when you’re dealing with real-world data issues.

Jane: It’s interesting because they also looked at batch size and found that making the batch size bigger helped boost the performance of that noisy automatically curated model more than it did for the manually labeled one. So, it suggests that as long as you use all that data, you can actually fight off some of those labeling errors with better training settings.

Tom: It proves that using an automated curation pipeline like MADLaP is a practical way to build high-performing deep learning models in medical imaging, even if the resulting data isn't perfectly clean. The authors didn’t claim perfection, they just showed improvement against their baseline.

Jane: And what this changes for us is that it gives us a clear path for developing AI tools without needing years of manual annotation time from specialists for every single study we want to incorporate into a model. It’s about making the development pipeline faster and more scalable.

Tom: So, the practical implication is clear: if you have access to an automated tool that can generate data like this, you should use everything it produces rather than just trying to filter down to the cleanest labels available.

Jane: And that leads us into a really important question for deployment—how do we handle that noise when we actually put this model in a hospital setting where things get messy and unpredictable?

The paper's improvements: Tom: We just talked about how using the full MADLaP set beats just using a small, super accurate subset for training this AI model, and now we’re looking at what they suggest as the next steps for actually making these results useful in a real workflow.

Jane: Right. The paper isn't just stopping at showing that one set is better than another; it's pushing forward with a few key ideas about how to handle the data structure itself and how to train the model effectively.

Tom: One major suggestion is that you need to use a large batch size when training on those automatically curated datasets, because they found that a larger batch size substantially increased performance over the manually selected set, even though the data was noisier.

Jane: That makes sense for production, because in real-world medical imaging, you often have to trade off speed for accuracy or you end up with very unstable training. So increasing that batch size seems like a way to manage that inherent noise in the data pipeline.

Tom: And the authors are really highlighting the utility of their automated tool, MADLaP, because they say it’s not just a nice experiment; it’s the only tool they have that can automatically annotate these thyroid datasets with rich information from pathology reports and radiology reports.

Jane: I agree. That's a big deal for accessibility in medical research. It moves the barrier away from needing every radiologist to manually label every image, which is what makes these studies so slow right now.

Tom: Lalam confirms this direction because based on the literature, I'm fairly sure this approach of using automated pipelines to generate large-scale annotated data is the most direct way to improve accuracy in classification tasks like this.

Jane: So where does that leave us? Since they proved that using all the data works better than just a clean subset, what’s the next challenge for anyone trying to deploy this kind of AI into a clinical setting?

Conclusion: Tom: So we’re wrapping up this look at the paper, "Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning." Basically, they proved that you should use all the data from that automated curation pipeline instead of just a small subset for training your classifier.

Jane: Exactly. They showed that using the full set gave them a significantly better performance score than using only the highly accurate subset, even though the full set had some noise in it.

Tom: It’s an interesting finding because it suggests that automated tools like MADLaP are genuinely beneficial for developing these kinds of high-performing deep learning models in medical imaging, provided you use all of that data.

Jane: It really changes how we think about building these systems—it points toward using automated pipelines to get large datasets without needing massive manual annotation efforts from specialists every time.

Lu: From a representation learning standpoint, it confirms that the quality of the input data, even with some noise, can be leveraged effectively if you use enough of it. It tells us that noisy data handling through larger batch sizes is a practical way to get better results in these specific classification tasks.

Meng: But I still have to ask about deployment costs; running a model trained on this full set versus the smaller set might have different inference times, and we need to know what breaks when you scale it up in a real hospital environment.

Lalam: I can confirm that the paper is very confident in its conclusion because it directly links the MADLaP pipeline to a measurable performance increase, which is exactly what we’re looking for when evaluating these methods.

Tom: So, to summarize, using an automatically curated dataset like the one from MADLaP substantially increases performance for thyroid nodule classification algorithms if you utilize all of that data points rather than just a small accurate subset.

Jane: It’s a solid result because it validates the idea that automated curation is a practical way to tackle the massive data hurdles in medical imaging.

Tom: Yeah, it’s about moving past manually curated sets for training and instead trusting the whole output of these automated tools.

Lu: This study opens up new avenues for how we design training procedures where you can leverage large volumes of noisy, but comprehensive, data efficiently.

Meng: I still think we need more work on how to reliably deploy this when it hits real-time clinical traffic because the performance gains are great in a lab setting, but scaling that robustness is another hurdle.

Lalam: The literature strongly supports the idea that this method can significantly improve classification accuracy for thyroid nodules.