UGV-based multimodal RGBD–multispectral fusion framework enables high-quality 3D phenotyping of greenhouse lettuce seedlings

Abstract

High-throughput phenotyping of lettuce seedlings is highly prone to background confusion because the seedlings are small, have weak textural features, and exhibit spectral reflectance similar to that of the substrate. Traditional single-visual-modality approaches struggle to achieve reliable structural and physiological characterization simultaneously under the repetitive backgrounds and dense arrangements typical of greenhouse tray cultivation. To address these challenges, we establish a multimodal 3D phenotyping framework tailored for controlled agriculture environments, enabling the quantification of structural and physiological characteristics of lettuce seedlings. This framework is based on an unmanned ground vehicle (UGV) platform integrating a RGBD camera and a quad-band multispectral sensor which are rigidly coupled and synchronously triggered. An alignment module based on established feature matching algorithm is introduced to register the misalignment between source multispectral and RGBD images. Subsequently, we design a novel dual-backbone instance segmentation network, MS-SegNet, to enhance segmentation accuracy by hierarchically fusing geometric information with multispectral features. A robust 3D metric pose estimation pipeline, incorporating standard SfM initialization, scale recovery, and generalized ICP refinement, is constructed to generate 3D point clouds with spectral attributes and semantic labels. Finally, key structural and physiological phenotype parameters of each seedling are calculated based on the 3D semantic multispectral point clouds. Experiments demonstrate that MS-SegNet achieves significant advantages in instance segmentation of lettuce seedlings with mAP@50:95 = 0.854. The metric 3D pose estimation pipeline exhibits reliable performance under complex controlled conditions. The quality of the 3D reconstructions is indirectly validated through downstream structural trait extraction. The estimated seedling height and crown width show high correlation with manual measurements, achieving R 2 values of 0.8379 and 0.918, and RMSE values of 10.94 mm and 11.56 mm, respectively. Overall, by systematically integrating these adapted components with the novel segmentation architecture, this framework achieves stable performance improvements in 3D reconstruction, instance segmentation, and phenotypic analysis under greenhouse conditions. It provides a scalable, integrated technical solution for non-destructive, high-throughput phenotyping of crop seedlings in controlled environments.

Plant phenotyping relevance

RGBD・マルチスペクトル・UGVを統合した3Dフェノタイピング基盤を開発し、分割・再構成・構造／生理形質抽出を検証しており、フェノタイピング手法が研究の中心である。

Code and data availability

The paper's multimodal RGBD–multispectral lettuce seedling dataset and the MS-SegNet/3D reconstruction analysis code are not publicly deposited; the authors state data are available only on request. Sensor vendor URLs are hardware pages, not paper-specific assets.

No evidence-backed public reproduction asset is currently recorded.