Plaque instability may lead to chronic embolization, which in turn may contribute to progressive cognitive decline. Accumulated strain tensor indices over a cardiac cycle within a pulsating carotid plaque may be viable biomarkers for the diagnosis of plaque instability. Using plaque-only carotid artery segmentations, we recently demonstrated that impaired cognitive function correlated significantly with maximum axial and lateral strain indices within a localized region of interest in plaque. Inclusion of the adventitial layer focuses our strain or instability measures on the vessel wall-plaque interface hypothesized to be a region with increased shearing forces and measureable instability. A hierarchical block-matching motion tracking algorithm developed in our laboratory was used to estimate accumulated axial, lateral, and shear strain distribution in plaques identified with the plaque-with-adventitia segmentation. Correlations of strain indices to the Repeatable Battery for the Assessment of Neuropsychological Status Total score were performed and compared with previous results. Overall, correlation coefficients (r) and significance (p) values improved for axial, lateral, and shear strain indices. Shear strain indices, however, demonstrated the largest improvement. The Pearson correlation coefficients for maximum shear strain and cognition improved from the previous plaque-only analyses of -0.432 and -0.345 to -0.795 and -0.717 with the plaque-with-adventitia segmentation for the symptomatic group and for all patients combined, respectively. Our results demonstrate the advantage of including adventitia for ultrasound carotid strain imaging providing improved association to parameters assessing cognitive impairment in patients. This supports theories of the importance of the vessel wall plaque interface in the pathophysiology of embolic disease.