Hyperspectral and multispectral imaging capture an expanded dimension of information that facilitates discoveries. However, image features are frequently obscured by noise generated from the limited photodamage threshold of the specimen. Although machine learning approaches demonstrate considerable promise in addressing this challenge, they typically require extensive datasets, which can be difficult to obtain. Here, we introduce BiFormer denoising network (BDN), designed to effectively and efficiently extract image features by utilizing both local and global level connections, sparse architectures, and fine-tuning. Experimental results indicate that BDN enhances the quality of stimulated Raman scattering (SRS) images by up to 16-fold in signal-to-noise ratio (SNR), particularly improving subtle features at higher spatial frequencies. Furthermore, BDN is successfully adapted to fluorescence imaging, achieving significant improvements in SNR and order-of-magnitude reduction in exposure time, thereby showcasing its versatility across various imaging modalities. Collectively, BDN exhibits substantial potential for spectroscopic imaging applications in the fields of biomedicine and materials science.