Idiopathic ventricular arrhythmias (VAs) are ventricular tachycardias (VTs) or premature ventricular contractions (PVCs) with a mechanism that is not related to myocardial scar. The sites of successful catheter ablation of idiopathic VA origins have been progressively elucidated and include both the endocardium and, less commonly, the epicardium. Idiopathic VAs usually originate from specific anatomical structures such as the ventricular outflow tracts, aortic root, atrioventricular (AV) annuli, papillary muscles, Purkinje network and so on, and exhibit characteristic electrocardiograms based on their anatomical background. Catheter ablation of idiopathic VAs is usually safe and highly successful, but can sometimes be challenging because of the anatomical obstacles such as the coronary arteries, epicardial fat pads, intramural and epicardial origins, AV conduction system and so on. Therefore, understanding the relevant anatomy is important to achieve a safe and successful catheter ablation of idiopathic VAs. This review describes the anatomical consideration in the catheter ablation of idiopathic VAs.