Integrated detection of citrus fruits and branches using a convolutional neural network

Abstract

The key technology for a fruit picking robot is to identify fruits in different occlusion states. Based on the mask regional convolutional neural network (Mask R-CNN) and a branch segment merging algorithm, an integrated system was developed to simultaneously detect and measure citrus fruits and branches. A training dataset was constructed for fruit and tree appearance, including single fruit, multiple fruits, occluded fruits, branches and trunk. A segmental labeling method for random and irregular branches is proposed to improve the precision of the Mask R-CNN. Based on the segmental mask regions identified by this model, a more precise bounding box is obtained by calculating the minimum enclosing rectangle of mask regions. Then, a branch segment merging algorithm reconstructs branches and the trunk. Diameters of fruits and branches are obtained by mapping the color image onto the depth image. The average precision of fruit and branch recognition are 88.15% and 96.27%, respectively. The average measurement error of fruits’ transverse diameters, fruits’ longitudinal diameters, and branch diameters are 2.52, 2.29, and 1.17 mm, respectively. Experiments show the detection system has good performance for all types of fruits and occlusions. This vision system can effectively help the robot to plan the appropriate picking path and avoid obstacles.

Plant phenotyping relevance

Mask R-CNNと深度画像を用いて果実・枝を検出し、直径を定量する画像ベースの植物形質計測システムが中心であり、単なる収穫対象の位置検出を超えている。

Code and data availability

公開本文の所在を確認できませんでした。非公開または購読が必要な可能性があります。

No evidence-backed public reproduction asset is currently recorded.