Phase retrieval is vital for quantitative x-ray phase contrast imaging. This work presents an iterative method to simultaneously retrieve the x-ray absorption and phase images from a single x-ray exposure. The proposed approach uses the photon-counting detectors’ energy-resolving capability in providing multiple spectrally resolved phase contrast images from a single x-ray exposure. The retrieval method is derived, presented, and experimentally tested with a multi-material phantom in an inline phase contrast imaging setup. By separating the contributions of photoelectric absorption and Compton scattering to the attenuation, the authors divide the phase contrast image into two portions, the attenuation map arises from photoelectric absorption and a pseudo phase contrast image generated by electron density. This way one can apply the Phase Attenuation Dualiby (PAD) algorithm and Fresnel propagation for the iteration. The retrieval results from the experimental images show that this iterative method is fast, accurate, robust against noise, and thus yields noticeable enhancement in contrast to noise ratios.