Abstract
The classification of extramural vascular invasion status using baseline magnetic resonance imaging in rectal cancer has gained significant attention as it is an important prognostic marker. Also, the accurate prediction of patients achieving complete response with primary staging MRI assists clinicians in determining subsequent treatment plans. Most studies utilised radiomics-based methods, requiring manually annotated segmentation and handcrafted features, which tend to generalise poorly. We retrospectively collected 509 patients from 9 centres, and proposed a fully automated pipeline for EMVI status classification and CR prediction with diffusion weighted imaging and T2-weighted imaging. We applied nnUNet, a self-configuring deep learning model, for tumour segmentation and employed learned multiple-level image features to train classification models, named MLNet. This ensures a more comprehensive representation of the tumour features, in terms of both fine-grained detail and global context. On external validation, MLNet, yielding similar AUCs as internal validation, outperformed 3D ResNet10, a deep neural network with ten layers designed for analysing spatiotemporal data, in both CR and EMVI tasks. For CR prediction, MLNet showed better results than the current state-of-the-art model using imaging and clinical features in the same external cohort. Our study demonstrated that incorporating multi-level image representations learned by a deep learning based tumour segmentation model on primary MRI improves the results of EMVI classification and CR prediction with good generalisation to external data. We observed variations in the contributions of individual feature maps to different classification tasks. This pipeline has the potential to be applied in clinical settings, particularly for EMVI classification.