Effective visual memory encoding, a function important for everyday functioning, relies on episodic and semantic memory processes. In patients with medial temporal lobe epilepsy (MTLE), memory deficits are common as the structures typically involved in seizure generation are also involved in acquisition, maintenance, and retrieval of episodic memories. In this study, we used group independent component analysis (GICA) combined with Granger causality analysis to investigate the neuronal networks involved in visual memory encoding during a complex fMRI scene-encoding task in patients with left MTLE (LMTLE; N = 28) and in patients with right MTLE (RMTLE; N = 18). Additionally, we built models of memory encoding in LMTLE and RMTLE and compared them with a model of healthy memory encoding (Nenert et al., 2014). For those with LMTLE, we identified and retained for further analyses and model generation 7 ICA task-related components that were attributed to four different networks: the frontal and posterior components of the DMN, visual network, auditory-insular network, and an "other" network. For those with RMTLE, ICA produced 9 task-related components that were attributed to the somatosensory and cerebellar networks in addition to the same networks as in patients with LMTLE. Granger causality analysis revealed group differences in causality relations within the visual memory network and MTLE-related deviations from normal network function. Our results demonstrate differences in the networks for visual memory encoding between those with LMTLE and those with RMTLE. Consistent with previous studies, the organization of memory encoding is dependent on laterality of seizure focus and may be mediated by functional reorganization in chronic epilepsy. These differences may underlie the observed differences in memory abilities between patients with LMTLE and patients with RMTLE and highlight the modulating effects of epilepsy on the network for memory encoding.