Tumor cells exist in close proximity with non-malignant cells. Extensive and multilayered crosstalk between tumor cells and stromal cells tailors the tumor microenvironment (TME) to support survival, growth, and metastasis. Fibroblasts are one of the largest populations of non-malignant host cells that can be found within the TME of breast, pancreatic, and prostate tumors. Substantial scientific evidence has shown that these cancer-associated fibroblasts (CAFs) are not only associated with tumors by proximity but are also actively recruited to developing tumors where they can influence other cells of the TME as well as influencing tumor cell survival and metastasis. This review discusses the impact of CAFs on breast cancer biology and highlights their heterogeneity, origin and their role in tumor progression, ECM remodeling, therapy resistance, metastasis, and the challenges ahead of targeting CAFs to improve therapy response.