Purpose Emerging evidence suggests physical activity (PA) is associated with cognitive function. To overcome limitations of self-report PA measures, this study investigated the association of accelerometer-measured PA with incident cognitive impairment and longitudinal cognition among older adults. Methods Participants were recruited from the cohort study Reasons for Geographic and Racial Differences in Stroke in the United States. Accelerometers provided PA measures, including the percentage of total accelerometer wearing time spent in moderate-to-vigorous-intensity PA (MVPA%), light-intensity PA, and sedentary time for four to seven consecutive days at baseline. Cognitive impairment was defined by the Six-Item Screener. Letter fluency, animal fluency, word list learning, and Montreal Cognitive Assessment (orientation and recall) were conducted to assess executive function and memory. Results Participants (N = 6452, 69.7 ± 8.5 yr, 55.3% women, 30.5% Black) with usable accelerometer and cognition measures spent extremely limited time in MVPA (1.5% ± 1.9% of accelerometer wearing time). During an average of 3 yr of follow-up, 346 cases of incident cognitive impairment were observed. After adjustments, participants in higher MVPA% quartiles had a lower risk of cognitive impairment (i.e., quartile 2: odds ratio = 0.64, 95% confidence interval = 0.48-0.84) and better maintenance in executive function (≥0.03 z-score units) and memory (≥0.12 z-score units) compared with quartile 1 (P < 0.05). Stratified analyses showed the same association among White adults, but higher MVPA% was associated with better maintenance of only memory among Black adults. No significance was found for light-intensity PA or sedentary time. Conclusion There was a dose-response relationship between MVPA% and cognitive function in older adults, with higher levels associated with a 36% or lower risk of cognitive impairment and better maintenance of memory and executive function over time, particularly in White adults.