Light field imaging can simultaneously record spatial and angular information of light signals to provide various computational imaging functions. However, traditional microlens array-based light field cameras usually suffer from a trade-off between spatial and angular resolutions. In contrast, focal scanning light field imaging (FSLFI) can digitally modulate an incident light field through an image stack captured at different focal planes and then utilize the transport-of-intensity property to computationally recover the full-resolution light field. This paper presents a unified light field reconstruction algorithm framework, which involves different types of algorithms, such as back-projection reconstruction and additive/multiplicative iterative reconstruction, for FSLFI. Based on the unified algorithm framework, we systematically analyze and investigate the FSLFI performance on noise sensitivity. Light fields are reconstructed at different noise levels to quantitatively analyze the FSLFI performances with different types of algorithms. Both simulation and actual experimental results demonstrate that the noise sensitivity and reconstruction accuracy are constrained by each other for FSLFI. Back-projection reconstruction is appropriate in high-efficiency light field reconstruction, while additive/multiplicative iterative reconstruction is suitable for high-accuracy light field imaging at high/low noise levels. These conclusions can apply to any FSLFI method covered by the unified algorithm framework, in which appropriate algorithms can be selected for high-quality light field imaging and measurement according to specific application scenarios.