The prefrontal cortex and limbic system are important components of the neural circuit that underlies stress and anxiety. These brain regions are connected by white matter tracts that support neural communication including the cingulum, uncinate fasciculus, and the fornix/stria-terminalis. Determining the relationship between stress reactivity and these white matter tracts may provide new insight into factors that underlie stress susceptibility and resilience. Therefore, the present study investigated sex differences in the relationship between stress reactivity and generalized fractional anisotropy (GFA) of the white matter tracts that link the prefrontal cortex and limbic system. Diffusion weighted images were collected and deterministic tractography was completed in 104 young adults (55 men, 49 women; mean age = 18.87 SEM = 0.08). Participants also completed self-report questionnaires (e.g., Trait Anxiety) and donated saliva (later assayed for cortisol) before, during, and after the Trier Social Stress Test. Results revealed that stress reactivity (area under the curve increase in cortisol) and GFA of the cingulum bundle varied by sex. Specifically, men demonstrated greater cortisol reactivity and greater GFA within the cingulum than women. Further, an interaction between sex, stress reactivity, and cingulum GFA was observed in which men demonstrated a positive relationship while women demonstrated a negative relationship between GFA and cortisol reactivity. Finally, trait anxiety was positively associated with the GFA of the fornix/stria terminalis – the white matter pathways that connect the hippocampus/amygdala to the hypothalamus. These findings advance our understanding of factors that underlie individual differences in stress reactivity.