In this paper, we present a mechanism called Markov Model Mediator (MMM) to facilitate the efficient and effective capturing of high-level image concepts in content-based image retrieval (CBIR). MMM serves as the retrieval engine of the CBIR system and uses affinity-based similarity measures. This mechanism is effective in capturing subjective user concepts in that it not only takes into consideration the global image features, but also learns the high-level concepts of the images from the history of user access patterns and access frequencies on the images in the image database, which differentiates it from the common methods in CBIR. The advantage of our proposed mechanism is that it exploits the richness in the structured description of visual contents as well as the relative affinity relationships among the images. Consequently, it provides the capability to bridge the gap between the low-level features and the high-level concepts. This mechanism is also efficient in that it integrates Principal Component Analysis (PCA) to significantly reduce the image search space at a low cost before performing exact similarity matching. An off-line training subsystem for this framework was implemented and integrated into our system. The experimental results demonstrate that MMM can effectively capture user's high-level concept more quickly. © Springer Science+Business Media, LLC 2007.