OBJECTIVE: To test the hypothesis that greater levels of leisure-time moderate to vigorous intensity physical activity (MVPA) in midlife or late life are associated with larger gray matter volumes, less white matter disease, and fewer cerebrovascular lesions measured in late life, we utilized data from 1,604 participants enrolled in the Atherosclerosis Risk in Communities study. METHODS: Leisure-time MVPA was quantified using a past-year recall, interviewer-administered questionnaire at baseline and 25 years later and classified as none, low, middle, and high at each time point. The presence of cerebrovascular lesions, white matter hyperintensities (WMH), white matter integrity (mean fractional anisotropy [FA] and mean diffusivity [MD]), and gray matter volumes were quantified with 3T MRI in late life. The odds of cerebrovascular lesions were estimated with logistic regression. Linear regression estimated the mean differences in WMH, mean FA and MD, and gray matter volumes. RESULTS: Among 1,604 participants (mean age 53 years, 61% female, 27% Black), 550 (34%), 176 (11%), 250 (16%), and 628 (39%) reported no, low, middle, and high MVPA in midlife, respectively. Compared to no MVPA in midlife, high MVPA was associated with more intact white matter integrity in late life (mean FA difference 0.13 per SD [95% confidence interval (CI) 0.004, 0.26]; mean MD difference -0.11 per SD [95% CI -0.21, -0.004]). High MVPA in midlife was also associated with a lower odds of lacunar infarcts (odds ratio 0.68, 95% CI 0.46, 0.99). High MVPA was not associated with gray matter volumes. High MVPA compared to no MVPA in late life was associated with most brain measures. CONCLUSION: Greater levels of physical activity in midlife may protect against cerebrovascular sequelae in late life.