Chemical Exchange Saturation Transfer (CEST) MRI is sensitive to dilute metabolites with exchangeable protons, allowing tissue characterization in diseases such as acute stroke and tumor. CEST quantification using multi-pool Lorentzian fitting is challenging due to its strong dependence on image signal-to-noise ratio (SNR), initial values and boundaries. Herein we proposed an Image Downsampling Expedited Adaptive Least-squares (IDEAL) fitting algorithm that quantifies CEST images based on initial values from multi-pool Lorentzian fitting of iteratively less downsampled images until the original resolution. The IDEAL fitting in phantom data with superimposed noise provided smaller coefficient of variation and higher contrast-to-noise ratio at a faster fitting speed compared to conventional fitting. We further applied the IDEAL fitting to quantify CEST MRI in rat gliomas and confirmed its advantage for in vivo CEST quantification. In addition to significant changes in amide proton transfer and semisolid macromolecular magnetization transfer effects, the IDEAL fitting revealed pronounced negative contrasts of tumors in the fitted CEST maps at 2 ppm and -1.6 ppm, likely arising from changes in creatine level and nuclear overhauser effects, which were not found using conventional method. It is anticipated that the proposed method can be generalized to quantify MRI data where SNR is suboptimal.