Dual- or multi-energy CT, also known as spectral CT, obtains x-ray attenuation measurements at two or more energy spectra, allowing quantification of materials with different compositions. This process is known as material decomposition, which is the basis for a number of spectral CT applications. The conventional image-domain basis material decomposition is based on a least-squares fitting between the underlying material-specific images and the measured source spectral CT images (i.e. energy-bin or energy-threshold CT images), and a non-iterative solution based on matrix inversion can be derived for this process. However, due to its ill-conditioned nature, the material decomposition process is intrinsically susceptible to noise amplification. Hence, material-specific images can be contaminated by the presence of strong noise, which compromises the conspicuity of small objects, and hinders the delineation of anatomical regions of interest and associated pathology. In this work, we describe an image domain material decomposition framework with prior knowledge aware iterative denoising (MD-PKAID). The proposed framework exploits the structural redundancy between the individual material-specific images and the source spectral CT images to retain structural details in denoised material-specific images. It directly treats material decomposition as a regularized optimization problem with spectral CT images measured with different energy spectra as inputs. Phantom, in vivo animal and human data were acquired on a research whole-body photon-counting-detector-based CT system and a dual-source, dual-energy CT system to test the proposed method. The phantom results show that the proposed MD-PKAID can reduce the root-mean-square-error of basis material quantification by 75% compared to the standard material decomposition based on matrix inversion, while preserving structural details and image resolution in the material-specific images. The initial in vivo results demonstrate that the proposed method can improve delineation of small vasculature features in iodine-specific images while reducing image noise.