Kim, Michael E., Gao, Chenyu, Newlin, Nancy R., Rudravaram, Gaurav, Krishnan, Aravind R., Ramadass, Karthik, Kanakaraj, Praitayini, Schilling, Kurt G., Dewey, Blake E., & Bennett, David Alan. (2025). “Scalable quality control on processing of large diffusion-weighted and structural magnetic resonance imaging datasets.” PLOS ONE, 20(8), e0327388. https://doi.org/10.1371/journal.pone.0327388
Careful quality control (QC) is essential when working with large medical imaging datasets, because poor-quality data can lead to wrong conclusions or poorly trained machine learning models. However, QC can be very time consuming. Most existing methods try to save time using automated tools that detect unusual data points, but these tools cannot catch every mistake. This means researchers still need to visually check the results of data processing in a reliable and scalable way.
In this study, we designed a QC pipeline for a large collection of brain scans, including diffusion-weighted and structural MRI. Our method was built to: (1) provide a consistent way for teams of researchers to perform and manage QC, (2) allow fast visualization of preprocessed data so the process is quicker without sacrificing quality, and (3) make it easy to combine and share QC results across datasets and pipelines.
We tested our method by comparing it to an automated QC approach on a set of 1,560 brain scans, and by measuring how much agreement there was between different researchers performing QC. The results showed mostly high agreement among researchers and only small differences compared to the automated method. Overall, while visual QC still takes time, our approach makes the process more streamlined and efficient.
Fig 1. Issues with automatic and team-based QC.
When maintaining large neuroimaging datasets with multiple processing pipelines, shallow quality control processes that rely on derived metrics can fail to catch instances of algorithmic failures. However, deep QC processes quickly become unscalable and inefficient as the amount of data available increases due to the required time for mass visualization of outputs. For example, opening 50,000 T1w images separately in an image viewer for deep QC can take over 60 hours if it takes five seconds to load images in and out of the viewer. Team driven efforts to alleviate such large time costs come with additional challenges due to inconsistencies in reporting and methods of performing QC.