Objectives: To examine the effects of racial residential segregation and structural racism on the diagnosis, treatment, and outcomes of patients with prostate cancer. Patients and Methods: This retrospective cohort study examined men diagnosed with prostate cancer between 2005 and 2015. We collected data from Black and White men, aged ≥30 years, living within the 100 most populous counties participating in the Surveillance, Epidemiology, and End Results programme, a nationally representative dataset. The racial Index of Dissimilarity, a validated measure of segregation, was the primary exposure of interest. Outcomes of interest included advanced stage at diagnosis (Stage IV), surgery for localised disease (Stage I–II), and 10-year overall and cancer-specific survival. Multivariable Poisson regression analyses with robust error variance estimated the relative risk (RR) of advanced stage at diagnosis and surgery for localised disease at differing levels of segregation. Survival analysis was performed using competing hazards analysis. Results: Multivariable models estimating stage at diagnosis showed that the disparities between Black and White men disappeared at low levels of segregation. Disparities in receiving surgery for localised disease persisted across all levels of segregation. In racially stratified analyses, segregation had no effect on stage at diagnosis or surgical resection for Black patients. White patients saw a 56% (RR 0.42, P < 0.001) reduced risk of presenting at advanced stage and 20% increased likelihood (RR 1.20, P < 0.001) of surgery for localised disease. Black patients in the lowest segregation areas had the lowest overall mortality, but the highest cancer-specific mortality. Conclusions: Our study provides evidence that residential segregation has a significant impact on Black–White disparities in prostate cancer, likely through improved outcomes for White patients and worse outcomes for Black patients in more segregated areas. These findings suggest that mitigating segregation and the downstream effects of socioeconomic factors could alleviate these disparities.