Behavior recognition of cage-free multi-broilers based on spatiotemporal feature learning

Yilei Hu; Jiaqi Xiong; Jinyang Xu; Zhichao Gou; Yibin Ying; Jinming Pan; Di Cui

doi:10.1016/j.psj.2024.104314

Behavior recognition of cage-free multi-broilers based on spatiotemporal feature learning

Poult Sci. 2024 Sep 13;103(12):104314. doi: 10.1016/j.psj.2024.104314. Online ahead of print.

Authors

Yilei Hu¹, Jiaqi Xiong¹, Jinyang Xu¹, Zhichao Gou¹, Yibin Ying¹, Jinming Pan¹, Di Cui²

Affiliations

¹ College of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou 310058, P. R. China; Key Laboratory of Intelligent Equipment and Robotics for Agriculture of Zhejiang Province, Hangzhou 310058, P. R. China.
² College of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou 310058, P. R. China; Key Laboratory of Intelligent Equipment and Robotics for Agriculture of Zhejiang Province, Hangzhou 310058, P. R. China. Electronic address: dicui@zju.edu.cn.

Abstract

Poultry behavior indicates their health, welfare, and production performance. Timely access to broilers' behavioral information can improve their welfare and reduce disease spread. Most behaviors require a period of observation before they can be accurately judged. However, the existing approaches for multi-object behavior recognition were mostly developed based on a single-frame image and ignored the temporal features in videos, which led to misrecognition. This study proposed an end-to-end method for recognizing multiple simultaneous behavioral events of cage-free broilers in videos by Broiler Behavior Recognition System (BBRS) based on spatiotemporal feature learning. The BBRS consisted of 3 main components: the improved YOLOv8s detector, the Bytetrack tracker, and the 3D-ResNet50-TSAM model. The basic network YOLOv8s was improved with MPDIoU to identify multiple broilers in the same frame of videos. The Bytetrack tracker was used to track each identified broiler and acquire its image sequence of 32 continuous frames as input for the 3D-ResNet50-TSAM model. To accurately recognize behavior of each tracked broiler, the 3D-ResNet50-TSAM model integrated a temporal-spatial attention module for learning the spatiotemporal features from its image sequence and enhancing inference ability in the case of its image sequence less than 32 continuous frames due to its tracker ID switching. Each component of BBRS was trained and tested with the rearing density of 7 to 8 birds/m². The results demonstrated that the mAP@0.5 of the improved YOLOv8s detector was 99.50%. The Bytetrack tracker achieved a mean MOTA of 93.89% at different levels of occlusion. The Accuracy, Precision, Recall, and F1score of the 3D-ResNet50-TSAM model were 97.84, 97.72, 97.65, and 97.68%, respectively. The BBRS showed satisfactory inference ability with an Accuracy of 93.98% when 26 continuous frames of the tracked broiler were received by the 3D-ResNet50-TSAM model. This study provides an efficient tool for automatically and accurately recognizing behaviors of cage-free multi-broilers in videos. The code will be released on GitHub (https://github.com/CoderYLH/BBRS) as soon as the study is published.

Keywords: behavior recognition; broiler; computer vision; end-to-end; spatiotemporal feature.