Published 2017. This article has been contributed to by US Government employees and their work is in the public domain in the USA. Introduction: Several findings suggest that testosterone (T) is neuroprotective and that declining T levels during aging are associated with cognitive and brain pathologies; however, little is known on T and brain health in middle-age. We examined the relationships of total T, bioavailable T, and sex hormone binding globulin (SHBG) levels with total and regional gray matter (GM) and white matter (WM) volumes in middle-aged men. We also evaluated the association of sex hormone levels with cognitive function. Methods: Analysis included 267 community-dwelling men participating in the Coronary Artery Risk Development in Young Adults (CARDIA) brain magnetic resonance imaging (MRI) substudy. Total T, bioavailable T, and SHBG levels were measured at three times from the 2nd to 4th decade of life; brain volumes were measured at the ages of 42–56. Associations were estimated using linear regression models, adjusted for several potential confounders. Results: Higher SHBG levels were associated with greater total WM volume (+3.15 cm3 [95% confidence interval [CI] = 0.01, 6.28] per one standard deviation higher SHBG). Higher SHBG levels were associated with lower total and regional GM volumes overall and significantly with smaller parietal GM volume (−0.96 cm3 [95%CI = −1.71, −0.21]). T levels were not related to brain volumes. Neither T nor SHBG levels were associated with cognitive function. Conclusion: Results suggest a role for SHBG in structural brain outcomes in men and emphasize the value of investigating SHBG levels as modulators of sex hormone and metabolic pathways regulating brain and behavioral characteristics in men.