Pathology image AI analysis software refers to independent software based on digital pathology images that utilizes artificial intelligence technologies such as deep learning for functions like segmentation and detection. It falls under the Class Ⅲ of medical device management. Digital pathology images include microscopic images of cells or tissues obtained through imaging devices, as well as pathology images from Whole Slide Imaging (WSI) technology. The software is used in medical institutions and/or medical laboratories to assist pathologists in providing information for disease diagnosis, prognosis, treatment, etc. However, it cannot be the sole basis for clinical diagnostic decisions.
The “Key Points for Performance Evaluation and Review of Pathology Image AI Analysis Software” aims to guide applicants in preparing and writing the non-clinical evaluation section of the registration application for pathology image AI analysis software. It also serves as a reference for technical review departments, focusing on requirements for software research data, including demand specifications and algorithm research data.
Requirements specifications consider data collection, algorithm performance, and usage restrictions. Data collection needs to consider the compliance, adequacy and diversity of data sources, the scientificity and rationality of data distribution, and the adequacy, validity and accuracy of data quality control. Data must come from different regions and no less than 3 home institution. Institutions should follow the procedures described in the manual for slide production, tissue staining, and immunohistochemistry preparation. Algorithm performance, considering the intended use of the product, should comprehensively assess the applicability and requirements of performance indicators such as analysis speed, sensitivity, specificity, repeatability, reproducibility, and generalization. Factors affecting algorithm performance, such as gradient vanishing, gradient explosion, overfitting, and underfitting, should also be considered. Usage restrictions need to accurately state scenarios where the product is prohibited or cautioned against, describe product usage scenarios, and provide necessary warning information.
Algorithm research data should include algorithm research reports for each artificial intelligence algorithm or algorithm combination. This includes basic algorithm information, algorithm risk management, algorithm requirements specifications, data collection, algorithm training, algorithm performance evaluation, and algorithm traceability analysis.
The security level of such software is categorized as severe. For data collection, a declaration of data source compliance is required, specifying information such as the names of data source institutions, their locations, data collection volumes, and ethical approval numbers. Data collection should be accompanied by a data collection operating procedure document, including data collection plans and standard operating procedures; clinical institutions primarily carry out data collection, and the collection process should involve the encryption and numbering of sample data, with the plan outlining numbering rules.
Data organization should clearly define data cleaning/preprocessing procedures, briefly describe the software used in data processing, and submit research data for each software used in data processing. Data labeling should specify the qualification requirements and training content for labeling personnel and arbitrators; labeling personnel and arbitrators should be pathologists, and data should be labeled by at least two individuals. A certain proportion of data can be selected for the evaluation of non-labeling personnel. Data set construction should clarify the methods and basis for dividing each data set, and samples from the training set, tuning set (if any), and test set should be non-overlapping and verified for duplicates.
Algorithm training needs to provide data distribution on disease composition in training sets and tuning sets (if any) based on the applicable population, data source institutions, collection equipment, sample types and other factors. Training and tuning should be based on the training set and tuning set, with clear specifications for evaluation indicators, training methods, training goals, and tuning methods. Algorithm performance evaluation needs to be based on the test set to evaluate the algorithm design and confirm the efficiency, sensitivity, and specificity of the software algorithm performance. The performance should meet the algorithm design requirements.
