Memory skepticism denies our memory beliefs could have any notable epistemic good. One route to memory skepticism is to challenge memory’s epistemic trustworthiness, that is, its functioning in a way necessary for it to provide epistemic justification. In this paper we develop and respond to this challenge. It could threaten memory in such a way that we altogether lack doxastic attitudes. If it threatens memory in this way, then the challenge is importantly self-defeating. If it does not threaten memory in this way, then the challenge leaves a foundation for an inference to the best explanation response, one we articulate and support.