Successful breast conserving surgery consists of complete removal of the tumor while sparing healthy surrounding tissue. Despite currently available imaging and margin assessment tools, recognizing tumor tissue at a resection margin during surgery is challenging. Diffuse reflectance spectroscopy (DRS), which uses light for tissue characterization, can potentially guide surgeons to prevent tumor positive margins. However, inter-patient variation and changes in tissue physiology occurring during the resection might hamper this light-based technology. Here we investigate how inter-patient variation and tissue status (in vivo vs ex vivo) affect the performance of the DRS optical parameters. In vivo and ex vivo measurements of 45 breast cancer patients were obtained and quantified with an analytical model to acquire the optical parameters. The optical parameter representing the ratio between fat and water provided the best discrimination between normal and tumor tissue, with an area under the receiver operating characteristic curve of 0.94. There was no substantial influence of other patient factors such as menopausal status on optical measurements. Contrary to expectations, normalization of the optical parameters did not improve the discriminative power. Furthermore, measurements taken in vivo were not significantly different from the measurements taken ex vivo. These findings indicate that DRS is a robust technology for the detection of tumor tissue during breast conserving surgery.