© 2018 The Author(s) The arginine methylation status of histones dynamically changes during many cellular processes, including hematopoietic stem/progenitor cell (HSPC) development. The arginine methyltransferases and the readers that transduce the histone codes have been defined. However, whether arginine demethylation actively occurs in cells and what enzyme demethylates the methylarginine residues during various cellular processes are unknown. We report that JMJD1B, previously identified as a lysine demethylase for H3K9me2, mediates arginine demethylation of H4R3me2s and its intermediate, H4R3me1. We show that demethylation of H4R3me2s and H3K9me2s in promoter regions is correlated with active gene expression. Furthermore, knockout of JMJD1B blocks demethylation of H4R3me2s and/or H3K9me2 at distinct clusters of genes and impairs the activation of genes important for HSPC differentiation and development. Consequently, JMJD1B−/− mice show defects in hematopoiesis. Altogether, our study demonstrates that demethylase-mediated active arginine demethylation process exists in eukaryotes and that JMJD1B demethylates both H4R3me2s and H3K9me2 for epigenetic programming during hematopoiesis. Li et al. identify the arginine demethylase (RDM) activity of JMJD1B, a known lysine demethylase (KDM). They reveal that JMJD1B actively mediates demethylation of histone markers H4R3me2s and H3K9me2 in hematopoietic stem/progenitor cells (HSPCs).