Background and aims: While numerous cross-sectional studies have demonstrated an inverse relationship between cardiorespiratory fitness (CRF) and carotid atherosclerosis in middle age, much less is known about the association of midlife CRF with carotid atherosclerosis in later life. Methods: We studied 1094 participants, free of cardiovascular disease, who completed a maximal exercise test (GXT) for an objective measure of CRF between ages 40 and 59 and carotid ultrasound after the age of 59, with at least five years between studies. Carotid intima media thickness was measured. Assessments were also made regarding the presence of plaque and percent stenosis in four regions: common carotid, bulb, internal carotid and external carotid arteries. Multivariable logistic regression models were constructed to estimate the association of CRF with carotid artery disease. Results: At the time of GXT and carotid scan, participants were aged 50.7 ± 5.7 years and 69.3 ± 6.4 years, respectively. Almost half of participants had high midlife CRF (48.6%); 41.3% and 10.1% had moderate and low CRF, respectively. Over a mean follow-up period of 18.6 ± 8.5 years, the odds of having carotid artery disease in later life in the high CRF group was 0.50 (95% CI: 0.29–0.87) compared with the low CRF group. Each 1 MET increase in CRF was associated with 10% lower odds of having carotid artery disease (OR = 0.89, 95% CI: 0.80–0.98). Conclusions: Midlife CRF was inversely associated with carotid artery disease measured almost two decades later. This may represent a mechanistic link between high midlife CRF and reduced risk of stroke in later life.