Among people living with HIV (PLWH), HIV-related stigma predicts nonadherence to antiretroviral therapy (ART); however, the role of stigma associated with drug use is largely unknown. We examined the association between substance use (SU) stigma and optimal ART adherence in a sample of 172 self-reported HIV-infected drug users. Participants completed surveys on SU, stigma, and ART adherence. The three substance classes with the greatest number of participants exhibiting moderate/high-risk scores were for cocaine/crack cocaine (66.28%), cannabis (64.53%), and hazardous alcohol consumption (65.70%). Multivariable logistic regression was conducted to investigate associations between levels of SU stigma and optimal ART adherence, adjusting for sociodemographic characteristics, severity of illicit drug use (alcohol, smoking and substance involvement screening test) and alcohol use severity (Alcohol Use Disorders Identification Test-C), HIV-related stigma, and social support. The odds of optimal adherence among participants experiencing moderate [Adjusted Odds Ratio (AOR) = 0.36, p = 0.039] and very high (AOR = 0.25, p = 0.010) levels of anticipated SU stigma were significantly lower than participants experiencing low levels of anticipated SU stigma. No other stigma subscales were significant predictors of ART adherence. Interventions aiming to improve ART adherence among drug-using PLWH need to address anticipated SU stigma.