Androgen receptor (AR) signaling is vital for the normal development of the prostate and is critically involved in prostate cancer (PCa). AR is not only found in epithelial prostate cells but is also expressed in various cells in the PCa-associated stroma, which constitute the tumor microenvironment (TME). In the TME, AR is expressed in fibroblasts, macrophages, lymphocytes and neutrophils. AR expression in the TME was shown to be decreased in higher-grade and metastatic PCa, suggesting that stromal AR plays a protective role against PCa progression. With that, the functionality of AR in stromal cells appears to deviate from the receptor's classical function as described in PCa cells. However, the biological action of AR in these cells and its effect on cancer progression remains to be fully understood. Here, we systematically review the pathological, genomic and biological literature on AR actions in various subsets of prostate stromal cells and aim to better understand the consequences of AR signaling in the TME in relation to PCa development and progression.