The generation and structural characteristics of random speckle patterns impact the implementation and imaging quality of computational ghost imaging. Their modulation is limited by traditional electronic hardware. We aim to address this limitation using the features of an all-optical neural network. This work proposes a real-time target recognition system based on an all-optical diffraction deep neural network for ghost imaging. We use a trained neural network to perform pure phase modulation on visible light, and directly complete the target recognition task by detecting the maximum value of light intensity signals at different positions. We optimized the system by simulating the effects of parameters, such as the number of layers of the network, photosensitive pixel, unit area etc., on the final recognition performance, and the accuracy of target recognition reached 91.73%. The trained neural network is materialised by 3D printing technology and experiments confirmed that the system successfully performs real-time target recognition at a low sampling rate of 1.25%. It also verified the feasibility and noise resistance of the system in practical application scenarios.