Many cancers, including glioblastoma (GBM), have a male-biased sex difference in incidence and outcome. The underlying reasons for this sex bias are unclear but likely involve differences in tumor cell state and immune response. This effect is further amplified by sex hormones, including androgens, which have been shown to inhibit anti-tumor T cell immunity. Here, we show that androgens drive anti-tumor immunity in brain tumors, in contrast to its effect in other tumor types. Upon castration, tumor growth was accelerated with attenuated T cell function in GBM and brain tumor models, but the opposite was observed when tumors were located outside the brain. Activity of the hypothalamus-pituitary-adrenal gland (HPA) axis was increased in castrated mice, particularly in those with brain tumors. Blockade of glucocorticoid receptors reversed the accelerated tumor growth in castrated mice, indicating that the effect of castration was mediated by elevated glucocorticoid signaling. Furthermore, this mechanism was not GBM specific, but brain specific, as hyperactivation of the HPA axis was observed with intracranial implantation of non-GBM tumors in the brain. Together, our findings establish that brain tumors drive distinct endocrine-mediated mechanisms in the androgen-deprived setting and highlight the importance of organ-specific effects on anti-tumor immunity.