Fine-grained 3D rice phenotyping via multi-scale NeRF and multimodal segmentation.

Abstract

Fine-grained 3D phenotypic analysis of rice plays a vital role in rice breeding and yield estimation. However, a comprehensive rice data acquisition and segmentation pipeline is still lacking. While Neural Radiance Fields (NeRF) have shown impressive results in crop-level 3D reconstruction, their high sensitivity to data volume and camera viewpoints often leads to reconstruction failures for rice. In addition, the large-scale rice point clouds, coupled with heavy occlusion and visual similarity among grains, pose significant challenges for fine-grained trait extraction. To address the challenge of reconstructing rice point clouds under low-quality data conditions, we propose a novel method named Multi-Scale NeRF(MSNeRF). This method incorporates a structure-detail collaborative reconstruction mechanism and a dynamic initialization density scheduling strategy. Furthermore, we introduce a multimodal and multitask rice dataset (MMR) as a benchmark resource for future research. For rice point cloud segmentation, we develop Vision Rice Knowledge Graph Network(VRKGNet), which comprises an image segmentation module, a projection module, and a point cloud segmentation module enhanced with a Transformer to enlarge the receptive field. VRKGNet performs standalone point cloud segmentation and integrates image segmentation results from multiple viewpoints as prior knowledge to enhance semantic and instance-level segmentation. Extensive experiments demonstrate that MSNeRF achieves high-fidelity point cloud reconstruction with as few as 10 viewpoints. VRKGNet achieves superior rice plant segmentation with a semantic segmentation mIoU of 88.79% and an instance segmentation AP 25 of 84.55%, outperforming mainstream algorithms.

Plant phenotyping relevance

米の3D形質取得・再構成・分割を中核とする手法開発であり、データセット／ベンチマークも提供しているため、植物フェノタイピング手法文献に該当する。

Code and data availability

The paper's authors explicitly state that the source code for MSNeRF and VRKGNet is publicly available on GitHub with testing scripts and test cases to reproduce the main results. The MMR dataset itself is only available upon request from the corresponding author, so it does not qualify as a public asset.