Male breast cancer is a rare disease, of which little is known. In contrast to female breast cancer, the very vast majority of all cases are positive for estrogen receptor alpha (ERα), implicating the function of this steroid hormone receptor in tumor development and progression. Consequently, adjuvant treatment of male breast cancer revolves around inhibition of ERα. In addition, the androgen receptor (AR) gradually receives more attention as a relevant novel target in breast cancer treatment. Importantly, the rationale of treatment decision making is strongly based on parallels with female breast cancer. Yet, prognostic indicators are not necessarily the same in breast cancer between both genders, complicating translatability of knowledge developed in female breast cancer toward male patients. Even though ERα and AR are expressed both in female and male disease, are the genomic functions of both steroid hormone receptors conserved between genders? Recent studies have reported on mutational and epigenetic similarities and differences between male and female breast cancer, further suggesting that some features are strongly conserved between the two diseases, whereas others are not. This review critically discusses the recent developments in the study of male breast cancer in relation to ERα and AR action and highlights the potential future studies to further elucidate the genomic regulation of this rare disease.