Study Objectives: Sleep duration has been associated with risk of dementia and stroke, but few studies have investigated the relationship between sleep duration and brain MRI measures, particularly in middle age. Methods: In a prospective cohort of 613 black and white adults (mean age = 45.4 years) enrolled in the Coronary Artery Risk Development in Young Adults (CARDIA) study, participants reported typical sleep duration, dichotomized into moderate sleep duration (> 6 to ≤ 8 h) and short sleep duration (≤ 6 h) at baseline (2005-2006). Five years later, we obtained brain MRI markers of white matter including fractional anisotropy, mean diffusivity, and white matter hyperintensities. Results: Compared to moderate sleepers, short sleepers had an elevated ratio of white matter hyperintensities to normal tissue in the parietal region (OR = 2.31, 95% CI: 1.47, 3.61) adjusted for age, race/sex, education, hypertension, stroke/TIA, depression, smoking status, and physical activity. White matter diffusivity was also higher, approximately a 0.2 standard deviation difference, in frontal, parietal, and temporal white matter regions, among those reporting shorter sleep duration in (P < 0.05 for all). Conclusions: Short sleep duration was associated with worse markers of white matter integrity in midlife. These mid-life differences in white matter may underlie the link between poor sleep and risk of dementia and stroke.