Multimodal learning on RGB-D image for precise litchi phenotyping and weight estimation

Abstract

Accurate measurement of key phenotypic traits, including the horizontal and vertical diameters, the weights of both fruit and pit, is essential for the selection of elite litchi cultivars and the advancement of breeding research. Manual measurement, however, is laborious, inefficient, and subjective, highlighting the urgent need for automated and precise phenotyping tools. Unlike apples, mangoes, and grapes, litchi combines a spiny, highly variable pericarp (heterogeneous areoles/tubercles across cultivars) with diverse seed morphology (including irregular, wrinkled aborted seeds), thereby increasing the difficulty of semantic segmentation and biasing diameters and weight estimation. This study presents LitchiPhenoNet, a multimodal learning framework for litchi phenotypic analysis that employs a dual-branch architecture integrating RGB (color/texture) and depth (spatial/structural) information. Experiments were conducted on an RGB-D dataset comprising 1,198 image pairs (1280×720) across 10 cultivars, using a stratified train/test split of 958/240 pairs by cultivar. To address inherent semantic and scale inconsistencies between modalities, the framework incorporates the RD-Fusion module for precise cross-modal feature extraction, improving robustness under complex and variable pericarp surfaces. Comparative experiments show that LitchiPhenoNet consistently outperforms leading YOLO-based models, achieving millimeter-level diameter estimation with coefficients of determination approaching 0.98 and mean errors within 2 mm. For weight estimation, gram-level precision is attained across whole fruit, pit, and pulp, with coefficients of determination up to 0.98 and mean errors comparable to repeated manual measurements. By handling fine-scale surface relief and cross-cultivar variability, the framework is readily extensible to other textured fruits and scalable for high-throughput phenotyping in breeding programs. Collectively, these results demonstrate that LitchiPhenoNet provides an efficient, reliable, and accurate solution for quantifying litchi phenotypic traits, substantially advancing the objectivity and efficiency of phenotypic analysis and breeding selection.

Plant phenotyping relevance

RGB-D画像を用いてライチ果実・種子・果肉の径と重量を自動推定する専用フレームワークを開発し、複数品種・比較実験で性能検証しているため、植物表現型取得法が中心である。

Code and data availability

The paper describes a litchi RGB-D phenotyping dataset and the LitchiPhenoNet model, but the Data Availability Statement explicitly states no datasets were generated or analysed, and no public code, model, or data URL is provided. The only URL present (ultralytics/ultralytics) is a cited third-party library, not a this

No evidence-backed public reproduction asset is currently recorded.