Single-photon imaging is an emerging technology in sensing that is capable of imaging and identifying remote objects under extreme conditions. However, it faces several challenges, such as low resolution and high noise, to do the task of object detection. In this work, we propose an enhanced You Only Look Once network to identify and localize objects within images generated by single-photon sensing. We then experimentally test the proposed network on both the self-built single-photon dataset and the VisDrone2019 public dataset. Our results show that our network achieves a higher detection accuracy than the baseline models. Moreover, it admits a higher average precision in detecting small single-photon objects. Our work is expected to aid significant progress in exploring practical applications of single-photon sensing.