Cancer-associated fibroblasts (CAFs) are abundantly present in the microenvironment of virtually all tumors and strongly impact tumor progression. Despite increasing insight into their function and heterogeneity, little is known regarding the origin of CAFs. Understanding the origin of CAF heterogeneity is needed to develop successful CAF-based targeted therapies. Through various transplantation studies in mice, we show that CAFs in both invasive lobular breast cancer and triple-negative breast cancer originate from mammary tissue-resident normal fibroblasts (NFs). Single-cell transcriptomics, in vivo and in vitro studies reveal the transition of CD26+ and CD26- NF populations into inflammatory CAFs (iCAFs) and myofibroblastic CAFs (myCAFs), respectively. Functional co-culture experiments show that CD26+ NFs transition into pro-tumorigenic iCAFs which recruit myeloid cells in a CXCL12-dependent manner and enhance tumor cell invasion via matrix-metalloproteinase (MMP) activity. Together, our data suggest that CD26+ and CD26- NFs transform into distinct CAF subpopulations in mouse models of breast cancer.