Higher levels of physical activity (PA) have been linked to better neurocognitive functioning in many populations. The current study examines the longitudinal association between PA and neurocognitive functioning among HIV-infected and HIV-uninfected persons. Community-dwelling adults (N = 291) self-reported level of PA and completed a comprehensive neuropsychological battery at two to four study visits (Mean follow-up time = 2.6 years). Participants were divided into three PA groups: “No PA” (no PA at any visit), “consistent PA” (PA at ≥50% of visits), and “inconsistent PA” (PA < 50% of visits). A mixed effect model, adjusting for significant covariates showed that all PA groups had statistically significant, yet modest, neurocognitive decline over time; and, the consistent PA group began with, and maintained, significantly better neurocognitive function compared to the other two PA groups. This effect was evident among both HIV-uninfected and HIV-infected persons, despite the fact that HIV-infected persons showed lower baseline neurocognitive function. PA is a modifiable lifestyle behavior that may help to protect against neurocognitive impairment regardless of HIV status, however, given the proportion of HIV-infected individuals who evidence neurocognitive difficulties, a focus on increasing PA seems warranted.