Background: Physical activity has been shown to reduce the risk of breast cancer-specific mortality. Although factors associated with physical inactivity in breast cancer survivors have been studied, a detailed examination at the population level is still lacking. Methods: We addressed this gap in 1236 women with a diagnosis of breast cancer from the 2016 Behavioral Risk Factor Surveillance System Cancer Survivorship module. Physical inactivity was defined as self-reported absence of leisure time physical activity. Factors examined in the multivariable logistic regression model included sociodemographic, behavioral factors, access to health care, health history, current cancer treatment, and pain from cancer or treatment. Results: Overall, older age (≥65 years: OR = 2.63, 95% CI: 1.25-5.55) and being underweight (BMI <18.5: OR = 6.11, 95% CI: 1.35-27.66), were identified as significant factors associated with physical inactivity. In models adjusting for sociodemographics (Model 1), and the prior plus behavioral factors (Model 2), pain from cancer or treatment was significantly associated with physical inactivity (Model 2: OR = 2.23, 95% CI: 1.16-4.28); however, after fully adjusting for all variables (Model 3), there was no longer evidence of a significant association between pain from cancer and physical activity in female survivors with breast cancer. Conclusions: We identified demographic (older age) and physical (low BMI and pain) factors to be significantly associated with physical inactivity among breast cancer survivors. Future interventions to promote physical activity in breast cancer survivors could benefit by taking into account these factors to develop tailored recommendations for increasing activity.