With the shift towards organ preserving treatment strategies in rectal cancer it has become increasingly important to accurately discriminate between a complete and good clinical response after neoadjuvant chemoradiotherapy (CRT). Standard of care imaging techniques such as CT and MRI are well equipped for initial staging of rectal tumors, but discrimination between a good clinical and complete response remains difficult due to their limited ability to detect small residual vital tumor fragments. To identify new promising imaging techniques that could fill this gap, it is crucial to know the size and invasion depth of residual vital tumor tissue since this determines the requirements with regard to the resolution and imaging depth of potential new optical imaging techniques. We analyzed 198 pathology slides from 30 rectal cancer patients with a Mandard tumor regression grade 2 or 3 after CRT that underwent surgery. For each patient we determined response pattern, size of the largest vital tumor fragment or bulk and the shortest distance from the vital tumor to the luminal surface. The response pattern was shrinkage in 14 patients and fragmentation in 16 patients. For both groups combined, the largest vital tumor fragment per patient was smaller than 1mm for 38% of patients, below 0.2mm for 12% of patients and for one patient as small as 0.06mm. For 29% of patients the vital tumor remnant was present within the first 0.01mm from the luminal surface and for 87% within 0.5mm. Our results explain why it is difficult to differentiate between a good clinical and complete response in rectal cancer patients using endoscopy and MRI, since in many patients submillimeter tumor fragments remain below the luminal surface. To detect residual vital tumor tissue in all patients included in this study a technique with a spatial resolution of 0.06mm and an imaging depth of 8.9mm would have been required. Optical imaging techniques offer the possibility of detecting majority of these cases due to the potential of both high-resolution imaging and enhanced contrast between tissue types. These techniques could thus serve as a complimentary tool to conventional methods for rectal cancer response assessment.