Over the last few decades, considerable evidence shows that greater levels of aerobic exercise and cardiovascular fitness benefit cognitive performance. However, the degree to which free-living activity in community settings is related to cognitive performance remains unclear, particularly in older adults vulnerable to disability. Also, it is unknown whether the manner in which daily physical activity (PA) and sedentary time are accumulated throughout the day is associated with cognition. Cross-sectional associations between accelerometer-characterized PA and sedentary patterns and cognitive performance were examined in 1,274 mobility-limited older adults. Percent time spent in various bout lengths of PA (≥1, ≥2, and ≥5 min) and sedentary (≥1, ≥30, and ≥60 min) was defined as the number of minutes registered divided by total wear time × 100. Percent time was then tertiled for each bout length. Multiple linear regression models were used to estimate the associations between accelerometer bout variables and separate cognitive domains that included processing speed (Digit Symbol Coding; DSC), immediate and delayed recall (Hopkins Verbal Learning Test; HVLT), information processing and selective attention (Flanker), working memory (n-back), reaction time (switch and non-switch reaction time), and a composite score that averaged results from all cognitive tests. After adjusting for demographics, behavioral factors, and morbid conditions, more time spent in PA was associated with higher DSC for all bout lengths (p < 0.03 for all). Higher PA was associated with higher HVLT and global cognition scores but only for longer bout lengths (p < 0.05 for all). The association was largely driven by participants who spent the lowest amount of time performing activity while awake (p < 0.04). An inverse linear relationship was observed between total sedentary time and DSC (p = 0.02), but not for other measures of cognition. These results suggest that, while higher PA was associated with higher cognitive performance, PA’s association with memory was sensitive to bout duration. The time, but not the manner, spent in sedentary behaviors showed a minor association with executive function. Further research is warranted to characterize longitudinal changes in daily activity and sedentary patterns as potential biophysical markers of cognitive status in older adults.