Spectral diffusion is the phenomenon of random jumps in the emission wavelength of narrow lines. This phenomenon is a major hurdle for applications of solid state quantum emitters like quantum dots, molecules, or diamond defect centers in an integrated quantum optical technology. Here, we provide further insight into the underlying processes of spectral diffusion of the zero-phonon line of single nitrogen vacancy centers in nano-size diamond by using a novel method based on photon correlation interferometry. The method works although the spectral diffusion rate is several orders of magnitude higher than the photon detection rate and thereby improves the time resolution of previous experiments with nano-size diamond by 6 orders of magnitude. We study the dependency of the spectral diffusion rate on the excitation power, temperature, and excitation wavelength under off-resonant excitation. Our results bring insight into the mechanism of spectral diffusion and suggest a strategy to increase the number of spectrally indistinguishable photons emitted by diamond nanocrystals.