UAV-based monocular 3D panoptic mapping for fruit shape completion in orchard

Abstract

Accurate fruit shape reconstruction under real-world field conditions is essential for high-throughput phenotyping, sensor-based yield estimation, and orchard management. Existing approaches based on 2D imaging or explicit 3D reconstruction often suffer from occlusions, sparse views, and complex scene dynamics as a result of the plant geometries. This paper presents a novel UAV-based monocular 3D panoptic mapping framework for robust and scalable fruit shape completion in orchards. The proposed method integrates (1) Grounded-SAM2 for multi-object tracking and segmentation (MOTS), (2) photogrammetric structure-from-motion for 3D scene reconstruction, and (3) DeepSDF, an implicit neural representation, for completing occluded fruit geometries with a neural network. We furthermore propose a new MOTS evaluation protocol to assess tracking performance without requiring ground truth annotations. Experiments conducted in both controlled laboratory conditions and an operational apple orchard demonstrate the accuracy of our 3D fruit reconstruction at the centimeter level. The Chamfer distance error of the proposed shape completion method using the DeepSDF shape prior reduces this to the millimeter level, and outperforms the traditional method, while Grounded-SAM2 enables robust fruit tracking across challenging viewpoints. The approach is highly scalable and applicable to real-world agricultural scenarios, offering a promising solution to reconstruct complete fruits with visibility higher than 10% for precise 3D fruit phenotyping at a large scale under occluded conditions.

Plant phenotyping relevance

果実形状を対象とするUAV画像・3D再構成・形状補完法を開発し、実験で精度評価しており、植物表現型取得が研究の中心である。

Code and data availability

The paper's authors publicly release their UAV orchard video data, lab 3D apple scans, and analysis code via a GitHub repository explicitly stated in the text. A Zenodo deposit (10.5281/zenodo.15635994) is also mentioned for the data, but its URL is not among the allowed URLs, so only the GitHub asset is reported.