Obscured tree branches segmentation and 3D reconstruction using deep learning and geometrical constraints

Abstract

The shortage of agricultural labourers worldwide has left many groups unharvested and wasted, which motivates researchers worldwide to research fruit harvesting robots extensively. One of the major problems in fruit harvesting is to selectively avoid hard obstacles such as tree branches so that more optimal picking positions can be found and more fruits throughout the tree can be harvested. However, tree branches are often obscured in unstructured natural orchards and thus necessary branch reconstruction and recovery are required. The current branch reconstruction and recovery methods for harvesting robots focus on planar reconstruction with few occlusions while the existing 3D tree modelling methods are not optimised for harvesting purposes that require low computational cost and high localisation accuracy. This work presented a novel framework that reconstructs and recovers 3D obscured branches from planar images and depth maps captured by an RGB-D camera. The framework comprises three parts: branch segmentation using Unet++, branch reconstruction using Point2Skeleton and branch recovery using a novel obscured branch recovery (OBR) algorithm. Branch segmentation using Unet++ with InceptionV3 encoder shows the best overall result with IoU and F1-score of 0.6249 and 0.7692 respectively. OBR recovery algorithm achieves average reconstruction accuracy of 0.72. The mean error of the reconstructed total surface and obscured surface using OBR is 18.68 mm and 38.11 mm with a standard deviation of 14.3 mm and 32.64 mm. The result shows that this framework can effectively reconstruct spatial information of visible and obscured branches from a single view image which can potentially be utilised in harvesting robots.

Plant phenotyping relevance

RGB-D画像から枝の分割・3D再構成・遮蔽部復元を行う技術が中心で、単なる収穫対象の位置検出を超えて植物器官の空間構造を定量化しているため、植物表現型計測法として含める。

Code and data availability

公開論文であることは確認できましたが、現在の公式API・許可済み取得経路では本文を自動取得できませんでした。

No evidence-backed public reproduction asset is currently recorded.