PlantSegNeRF: A few-shot, cross-species method for plant 3D instance point cloud reconstruction via joint-channel NeRF with multi-view image instance matching

Abstract

Organ segmentation of plant point clouds is a prerequisite for the high-resolution and accurate extraction of organ-level phenotypic traits. Although the fast development of deep learning has boosted much research on segmentation of plant point clouds, the existing techniques for organ segmentation still face limitations in resolution, segmentation accuracy, and generalizability across various plant species. In this study, we proposed a novel approach called plant segmentation neural radiance fields (PlantSegNeRF), aiming to directly generate high-precision instance point clouds from multi-view RGB image sequences for a wide range of plant species. PlantSegNeRF performed two-dimensional (2D) instance segmentation on the multi-view images to generate instance masks for each organ with a corresponding instance identification (ID). The multi-view instance IDs corresponding to the same plant organ were then matched and refined using a specially designed instance matching (IM) module. The instance NeRF was developed to render an implicit scene containing color, density, semantic and instance information, which was ultimately converted into high-precision plant instance point clouds based on volume density. The results proved that in semantic segmentation of point clouds, PlantSegNeRF outperformed the commonly used methods, demonstrating an average improvement of 16.1 %, 18.3 %, 17.8 %, and 24.2 % in precision, recall, F1-score, and intersection over union (IoU) compared to the second-best results on structurally complex datasets. More importantly, PlantSegNeRF exhibited significant advantages in instance segmentation. Across all plant datasets, it achieved average improvements of 11.7 %, 38.2 %, 32.2 % and 25.3 % in mean precision (mPrec), mean recall (mRec), mean coverage (mCov), and mean weighted coverage (mWCov), respectively. Furthermore, PlantSegNeRF demonstrates superior few-shot, cross-species performance, requiring only multi-view images of few plants to train models applicable to specific or similar varieties. This study extends organ-level plant phenotyping and provides a high-throughput way to supply high-quality 3D data for developing large-scale artificial intelligence (AI) models in plant science. • A comprehensive dataset of well-labeled two-dimensional (2D) images and point clouds dataset of plants was established, including various varieties and growth stages. 50 plant samples were collected for each type. • A novel multi-view image instance matching (IM) module was proposed to align plant organ instance identifications (IDs) across different viewpoints, serving as the foundation for organ-level instance segmentation. • A multi-channel instance neural radiance fields (NeRF) module with encoding color, semantic, and instance information was developed to achieve high-precision mapping of 2D image colors, semantics, and aligned instances into 3D space, enabling point cloud background removal and fine-grained segmentation of plant organs.

Plant phenotyping relevance

植物器官の3D点群再構成・インスタンス分割を開発し、セグメンテーション性能を検証する手法研究であり、器官レベル表現型抽出を直接支援するため。

Code and data availability

公開論文であることは確認できましたが、現在の公式API・許可済み取得経路では本文を自動取得できませんでした。

No evidence-backed public reproduction asset is currently recorded.