IPENS: Interactive unsupervised framework for rapid plant phenotyping extraction via NeRF-SAM2 fusion

Abstract

Advanced plant phenotyping technologies are vital for trait improvement and accelerating intelligent breeding. Due to the species diversity of plants, existing methods heavily rely on large-scale high-precision manually annotated data. For self-occluded objects at the grain level, unsupervised methods often prove ineffective. This study proposes IPENS, an interactive unsupervised multi-target point cloud extraction method. It utilizes radiance field information to lift 2D masks, segmented by SAM2 (Segment Anything Model 2), into 3D space for target point cloud extraction. A multi-target collaborative optimization strategy addresses the challenge of segmenting multiple targets from a single interaction. On a rice dataset, IPENS achieves a grain-level segmentation mean Intersection over Union (mIoU) of 63.72%. For phenotypic trait estimation, it achieves a grain voxel volume coefficient of determination R 2 = 0.7697 (Root Mean Square Error, RMSE = 0.0025), leaf surface area R 2 = 0.84 (RMSE = 18.93), and leaf length and width prediction accuracies of R 2 = 0.97 and R 2 = 0.87 (RMSE = 1.49 and 0.21). On a wheat dataset, IPENS further improves segmentation performance to a mIoU of 89.68%, with exceptional phenotypic estimation results: panicle voxel volume R 2 = 0.9956 (RMSE = 0.0055), leaf surface area R 2 = 1.00 (RMSE = 0.67), and leaf length and width predictions reaching R 2 = 0.99 and R 2 = 0.92 (RMSE = 0.23 and 0.15). Without requiring annotated data, IPENS rapidly extracts grain-level point clouds for multiple targets within three minutes using single-round image interactions. These features make IPENS a high-quality, non-invasive phenotypic extraction solution for rice and wheat, offering significant potential to enhance intelligent breeding.

Plant phenotyping relevance

植物形質抽出のためのNeRF-SAM2融合手法を開発し、作物データセットで分割性能と形質推定精度を検証しているため、方法開発・検証が中心である。

Code and data availability

The paper's analysis code is publicly available on GitHub. The rice/wheat MMR/MMW phenotype datasets (multi-view images, point clouds, annotations) are only available upon reasonable request, so they are not public.