The adult human breast comprises an intricate network of epithelial ducts and lobules that are embedded in connective and adipose tissue. While previous studies have mainly focused on the breast epithelial system, many of the non-epithelial cell types remain understudied. Here, we constructed a comprehensive Human Breast Cell Atlas (HBCA) at single-cell and spatial resolution. Our single-cell transcriptomics data profiled 535,941 cells from 62 women, and 120,024 nuclei from 20 women, identifying 11 major cell types and 53 cell states. These data revealed abundant pericyte, endothelial and immune cell populations, and highly diverse luminal epithelial cell states. Our spatial mapping using three technologies revealed an unexpectedly rich ecosystem of tissue-resident immune cells in the ducts and lobules, as well as distinct molecular differences between ductal and lobular regions. Collectively, these data provide an unprecedented reference of adult normal breast tissue for studying mammary biology and disease states such as breast cancer.