This study introduces a machine learning (ML) pipeline designed to assist in the preoperative differential diagnosis of leiomyosarcoma (LMS) from leiomyomas, which are often difficult to distinguish due to similar clinical presentations. The pipeline is innovative in several ways:

– It was developed with input from end-users, including healthcare professionals, to ensure its relevance and usability in clinical settings.
– It provides transparent decision-making, allowing clinicians to understand how the ML algorithms arrive at their conclusions and the significance of different data features in these decisions.
– It addresses the issue of class imbalance (where one diagnosis is much rarer than the other) by selecting the best combination of synthetic oversampling and classification algorithms.
– It offers explainability for its decisions at both a global and local level, with the SHAP algorithm quantifying feature impact for individual predictions.
– A natural language explanation module translates the ML decisions into easily interpretable reports for clinicians.

The pipeline demonstrated high performance, with an Area Under the Curve (AUC) of 0.99 and an F1 score of 0.87, indicating excellent accuracy and precision. Two ultrasound features—tumor borders and lesion consistency—were identified as having a significant impact on the model’s decisions. This work is important as it not only provides a tool for better diagnosis but also enhances trust and understanding of ML in healthcare.