3D Hierarchical Panoptic Segmentation in Real Orchard Environments Across Different Sensors

Abstract

Crop yield estimation is a relevant problem in agriculture, because an accurate yield estimate can support farmers' decisions on harvesting or precision intervention. Robots can help to automate this process. To do so, they need to be able to perceive the surrounding environment to identify target objects such as trees and plants. In this paper, we introduce a novel approach to address the problem of hierarchical panoptic segmentation of apple orchards on 3D data from different sensors. Our approach is able to simultaneously provide semantic segmentation, instance segmentation of trunks and fruits, and instance segmentation of trees (a trunk with its fruits). This allows us to identify relevant information such as individual plants, fruits, and trunks, and capture the relationship among them, such as precisely estimate the number of fruits associated to each tree in an orchard. To efficiently evaluate our approach for hierarchical panoptic segmentation, we provide a dataset designed specifically for this task. Our dataset is recorded in Bonn, Germany, in a real apple orchard with a variety of sensors, spanning from a terrestrial laser scanner to a RGB-D camera mounted on different robots platforms. The experiments show that our approach surpasses state-of-the-art approaches in 3D panoptic segmentation in the agricultural domain, while also providing full hierarchical panoptic segmentation. Our dataset is publicly available at https://www.ipb.uni-bonn.de/data/hops/. The open-source implementation of our approach is available at https://github.com/PRBonn/hapt3D.

Plant phenotyping relevance

リンゴ樹・果実・幹を3Dセグメンテーションし、樹ごとの果実数を推定する手法と専用データセットを中心に開発・評価しており、植物の器官形態・収量関連形質の取得に該当する。

Code and data availability

The paper introduces the HOPS dataset of annotated 3D apple orchard point clouds (TLS, UAV, UGV, SfM) for hierarchical panoptic segmentation, publicly available at the authors' IPB Bonn page, and releases the open-source implementation (hapt3D) on GitHub. Both are paper-specific, public, and actionable.