The objective was to examine bidirectional associations of accelerometer estimated sedentary time and physical activity with reported knee symptoms. Participants were 2,034 adults (mean age 45.3 ± 3.6 years, 58.7% female) from CARDIA. Generalized estimating equations for logistic regression and linear mixed regression models examined associations of accelerometer estimated sedentary time, light-intensity physical activity (LPA), and moderate-to-vigorous intensity physical activity (MVPA) at baseline (2005–06) with knee discomfort, pain, stiffness, and physical function (yes/no and continuous scores from short-form WOMAC function scale) at the 5- and 10-year follow-up exams. Linear regression models examined associations between knee symptoms at the 5-year follow-up with accelerometer estimates at the 10-year follow-up. Models were adjusted for confounders; individuals with comorbidities were excluded in sensitivity analyses. A 30 min/day increment in sedentary time at baseline was associated with lower odds of knee symptoms at the 5- and 10-year follow-up (OR: 0.95, 95% CI range: 0.92–0.98), while LPA and MVPA were associated with greater odds of knee symptoms (LPA OR range: 1.04–1.05, 95% CI range: 1.01–1.09; MVPA OR range: 1.17–1.19, 95% CI range: 1.06–1.32). Report of knee symptoms at the 5-year follow-up was associated with 13.52–17.51 (95% CI range: −29.90, −0.56) fewer minutes/day of sedentary time and 14.58–17.51 (95% CI range: 2.48, 29.38) more minutes/day of LPA at the 10-year follow-up, compared to those reporting no symptoms. Many associations were no longer statistically significant when excluding individuals with comorbidities. Findings support a bidirectional association of accelerometer estimated sedentary time and physical activity with knee symptoms across midlife.