Citrus pose estimation from an RGB image for automated harvesting

Abstract

Automated fruit harvesting is promising research in the development of agricultural modernization. However, the complex and non-structural orchard environment is extremely challenging. In order to meet the needs of different end-effectors and to improve the success rate of automatic fruit harvesting, it is critical to perform fruit pose estimation before picking operations. In this study, a citrus pose estimation method through a single RGB image is introduced. The rotation of the citrus pose is defined as a vector that passes through the center of the fruit, which is perpendicular to the plane where the fruit navel point is located. Simply speaking, a multi-task learning model named FPENet is proposed to simultaneously locate the fruit navel point and predict the fruit rotation vector. And a hyperparameter is introduced in the loss function to achieve the simultaneous convergence of multiple tasks. In addition, this paper designs a 2D image annotation tool and constructs a citrus pose dataset, which contributes to model training and also the algorithm evaluation. In the experiment, we evaluate and analyze each module of the proposed network structure, and verify its performance on a harvesting robot. The experimental results show that the FPENet achieves an 88.92 AP score on fruit navel point detection, and 11.13° on the average error of the rotation vector. Over 90% of rotation vectors have an angular error of less than 22.5°. The harvesting success rate is 79.79%. This study offers a new idea for fruit pose estimation and provides the possibility and foundation for estimating fruit pose with a 2D image input.

Plant phenotyping relevance

RGB画像から果実のへそ位置と回転ベクトル（果実姿勢）を推定する手法を開発し、データセットと注釈ツールも構築している。収穫対象の単なる検出・位置特定を超えて、再利用可能な果実器官の姿勢形質を抽出するため、中心的なフェノタイピング手法と判断する。

Code and data availability

公開状態または取得可能な本文経路を確認できませんでした。

No evidence-backed public reproduction asset is currently recorded.