SSL-NBV: A self-supervised-learning-based next-best-view algorithm for efficient 3D plant reconstruction by a robot

Abstract

The 3D reconstruction of plants is challenging due to their complex shape causing many occlusions. Next-Best-View (NBV) methods address this by iteratively selecting new viewpoints to maximize information gain (IG). Deep-learning-based NBV (DL-NBV) methods demonstrate higher computational efficiency over classic voxel-based NBV approaches but current methods require extensive training using ground-truth plant models, making them impractical for real-world plants. These methods, moreover, rely on offline training with pre-collected data, limiting adaptability in changing agricultural environments. This paper proposes a self-supervised learning-based NBV method (SSL-NBV) that uses a deep neural network to predict the IG for candidate viewpoints. The method allows the robot to gather its own training data during task execution by comparing new 3D sensor data to the earlier gathered data and by employing weakly-supervised learning and experience replay for efficient online learning. Comprehensive evaluations were conducted in simulation and real-world environments using cross-validation. The results showed that SSL-NBV required fewer views for plant reconstruction than non-NBV methods. It achieved IG prediction in 0.0038s, making it over 800 times faster than a voxel-based NBV, and an online learning iteration in 0.099s. SSL-NBV reduced training annotations by over 90% compared to a baseline DL-NBV. Furthermore, SSL-NBV could adapt to novel scenarios through online fine-tuning. Also using real plants, the results showed that the proposed method can learn to effectively plan new viewpoints for 3D plant reconstruction. Most importantly, SSL-NBV automated the entire network training and uses continuous online learning, allowing it to operate in changing agricultural environments.

Plant phenotyping relevance

植物の3D再構成に必要な視点選択を自己教師あり学習で開発し、実植物を用いて評価しているため、植物形態フェノタイピングの取得手法が中心です。

Code and data availability

The paper reports plant 3D reconstruction/NBV experiments with simulation and real-world point clouds, but no public dataset, code, or model repository is provided. The Data availability statement says data are available only on request, so no public paper-specific asset qualifies.

No evidence-backed public reproduction asset is currently recorded.