In estrogen-receptor-positive, HER2-negative (ER+HER2-) breast cancer, higher levels of tumor infiltrating lymphocytes (TILs) are often associated with a poor prognosis and this phenomenon is still poorly understood. Fibroblasts represent one of the most frequent cells in breast cancer and harbor immunomodulatory capabilities. Here, we evaluate the molecular and clinical impact of the spatial patterns of TILs and fibroblast in ER+HER2- breast cancer. We used a deep neural network to locate and identify tumor, TILs, and fibroblasts on hematoxylin and eosin-stained slides from 179 ER+HER2- breast tumors (ICGC cohort) together with a new density estimation analysis to measure the spatial patterns. We clustered tumors based on their spatial patterns and gene set enrichment analysis was performed to study their molecular characteristics. We independently assessed the spatial patterns in a second cohort of ER+HER2- breast cancer (N = 630, METABRIC) and studied their prognostic value. The spatial integration of fibroblasts, TILs, and tumor cells leads to a new reproducible spatial classification of ER+HER2- breast cancer and is linked to inflammation, fibroblast meddling, or immunosuppression. ER+HER2- patients with high TIL did not have a significant improved overall survival (HR = 0.76, P = 0.212), except when they had received chemotherapy (HR = 0.447). A poorer survival was observed for patients with high fibroblasts that did not show a high level of TILs (HR = 1.661, P = 0.0303). Especially spatial mixing of fibroblasts and TILs was associated with a good prognosis (HR = 0.464, P = 0.013). Our findings demonstrate a reproducible pipeline for the spatial profiling of TILs and fibroblasts in ER+HER2- breast cancer and suggest that this spatial interplay holds a decisive role in their cancer-immune interactions.