Spectrometer miniaturization has become a significant trend driven by the demand for distributed and continuous spectral sensing. Broadband encoding spectrometers, which utilize broadband encoder arrays to extract spectral features and algorithms to reconstruct spectra, are among the most competitive candidates for high-performance miniaturized spectrometers. Enhancing the spectral feature extraction capability of these broadband encoders is essential for improving spectrometer performance. However, the strategies and approaches for optimizing these encoders are not yet well-defined. This study analyzes the effectiveness of improving the basis orthogonality of the encoders for their optimization and proposes a dual-layer broadband encoder structure to implement this optimization strategy. The designed dual-layer broadband encoders consist of vertically stacked quantum dot encoders and TiO2/SiO2 encoders, with the corresponding basis being mixed Gaussian. Simulation experiments demonstrate that the proposed dual-layer broadband encoder structure significantly improves the encoders' basis orthogonality, leading to enhanced spectral detection accuracy of the spectrometer constructed with these dual-layer encoders. Experimental fabrication of the dual-layer encoders confirms their physical feasibility and basis orthogonality enhancement.