To examine associations between lipohypertrophy and lipoatrophy and illicit drug use, smoking, and at-risk alcohol use among a large diverse cohort of persons living with HIV (PLWH) in clinical care. 7,931 PLWH at six sites across the United States completed 21,279 clinical assessments, including lipohypertrophy and lipoatrophy, drug/alcohol use, physical activity level, and smoking. Lipohypertrophy and lipoatrophy were measured using the FRAM body morphology instrument and associations were assessed with generalized estimating equations. Lipohypertrophy (33% mild, 4% moderate-to-severe) and lipoatrophy (20% mild, 3% moderate-to-severe) were common. Older age, male sex, and higher current CD4 count were associated with more severe lipohypertrophy (p values <.001-.03). Prior methamphetamine or marijuana use, and prior and current cocaine use, were associated with more severe lipohypertrophy (p values <.001-.009). Older age, detectable viral load, and low current CD4 cell counts were associated with more severe lipoatrophy (p values <.001-.003). In addition, current smoking and marijuana and opiate use were associated with more severe lipoatrophy (p values <.001-.03). Patients with very low physical activity levels had more severe lipohypertrophy and also more severe lipoatrophy than those with all other activity levels (p values <.001). For example, the lipohypertrophy score of those reporting high levels of physical activity was on average 1.6 points lower than those reporting very low levels of physical activity (-1.6, 95% CI:-1.8 to-1.4, p <.001). We found a high prevalence of lipohypertrophy and lipoatrophy among a nationally distributed cohort of PLWH. While low levels of physical activity were associated with both lipohypertrophy and lipoatrophy, associations with substance use and other clinical characteristics differed between lipohypertrophy and lipoatrophy. These results support the conclusion that lipohypertrophy and lipoatrophy are distinct, and highlight differential associations with specific illicit drug use.