Lettuce architectural phenotypes extraction from multimodal images by low-light sensitivity and strong spatial perception

Abstract

Accurate measurement of structural phenotypes, such as plant height and canopy width, is crucial for the scientific management of lettuce cultivation in Plant Factories with Artificial Lighting (PFALs). In this study, we developed a multimodal image fusion model using visible images (RGB), depth images (Depth), and infrared images (IR) to extract lettuce phenotypes in PFAL environments. We proposed a Residual Space information enhancement module (DRS) and a fusion feature supplement method with adaptive weight optimization for IR features (IRC) to address the weak spatial perception of traditional RGB-based models and the feature loss of RGB due to illumination disturbance. Three lettuce varieties (Bixiao, Huqian, and Mondai) were selected as experimental subjects to evaluate the robustness of our proposed model. In ablation experiments, the benchmark model improved by DRS increased by 1.6% and 0.9% in terms of MAP0.75 and MAP0.5:0.95, respectively. The benchmark model improved by IRC increased by 0.2%, 0.6%, and 1.2% in terms of MAP0.5, MAP0.75, and MAP0.5:0.95, respectively. Furthermore, MAP0.5, MAP0.75, and MAP0.5:0.95 values increased by 0.3%, 3.3%, and 2.3% when the two modules were combined, respectively. Compared with manually measured plant height and canopy width, the Root Mean Square Error (RMSE) of the average plant height prediction results for the three varieties is 0.74, and the Mean Squared Error (MSE) is 0.55. For canopy width, the RMSE of the model’s prediction results was 0.70, and the MSE was 0.49. In the lighting influence experiment, our method outperformed the unimproved model by approximately 0.3–4% in terms of MAP0.5, MAP0.75, and MAP0.5:0.95 across multiple datasets. Our proposed model effectively addresses lighting disturbance, enhances the robustness of the baseline model against varying lighting conditions, improves spatial perception capability, facilitates the separation of adjacent plant features in the model’s feature extraction stage, enhances the model’s detection ability, and ultimately improves phenotype extraction capability.

Plant phenotyping relevance

マルチモーダル画像融合モデルを開発し、レタスの草丈・群落幅という植物表現型を画像から抽出・推定しており、手法開発と技術評価が研究の中心である。

Code and data availability

公開状態または取得可能な本文経路を確認できませんでした。

No evidence-backed public reproduction asset is currently recorded.