Pelvic organ prolapse (POP), downward descent of the pelvic organs resulting in a protrusion of the vagina, is a highly prevalent condition, responsible for 300,000 surgeries in the U.S. annually. Rectocele, a posterior vaginal wall (PVW) prolapse of the rectum, is the second most common type of POP after cystocele. A rectocele usually manifests itself along with other types of prolapse with multicompartment pelvic floor defects. To date, the specific mechanics of rectocele formation are poorly understood, which does not allow its early stage detection and progression prediction over time. Recently, with the advancement of imaging and computational modeling techniques, a plethora of finite element (FE) models have been developed to study vaginal prolapse from different perspectives and allow a better understanding of dynamic interactions of pelvic organs and their supporting structures. So far, most studies have focused on anterior vaginal prolapse (AVP) (or cystocele) and limited data exist on the role of pelvic muscles and ligaments on the development and progression of rectocele. In this work, a full-scale magnetic resonance imaging (MRI) based three-dimensional (3D) computational model of the female pelvic anatomy, comprising the vaginal canal, uterus, and rectum, was developed to study the effect of varying degrees (or sizes) of rectocele prolapse on the vaginal canal for the first time. Vaginal wall displacements and stresses generated due to the varying rectocele size and average abdominal pressures were estimated. Considering the direction pointing from anterior to posterior side of the pelvic system as the positive Y-direction, it was found that rectocele leads to negative Y-direction displacements, causing the vaginal cross section to shrink significantly at the lower half of the vaginal canal. Besides the negative Y displacements, the rectocele bulging was observed to push the PVW downward toward the vaginal hiatus, exhibiting the well-known "kneeling effect." Also, the stress field on the PVW was found to localize at the upper half of the vaginal canal and shift eventually to the lower half with increase in rectocele size. Additionally, clinical relevance and implications of the results were discussed.