TomPhenoNet: A multi-modal fusion and multi-task learning network model for monitoring growth parameters of dwarf tomatoes

Abstract

Dwarf tomatoes, with high edible and ornamental value, require monitoring multiple growth parameters to balance yield and aesthetics. While deep learning has been widely applied in phenotype monitoring, most studies focus on individual growth parameters, overlooking intrinsic relationships. To simultaneously monitor multiple growth parameters across the entire growth stage and different cultivars, this study develops a multi-modal multi-task phenotype monitoring network for dwarf tomatoes (TomPhenoNet). The network model utilizes top-view RGB-D images to evaluate four key growth parameters: height, leaf area, fresh weight, and the number of red fruits. TomPhenoNet generates mask images, fruit detection features, and the number of detected fruits based on RGB images. By fusing RGB-D images, mask images, and fruit detection features, and introducing the cross-stitch network, the network predicts plant height, leaf area, and fresh weight. The predicted values are further used to generate the dynamic occlusion coefficient, adjusting the number of detected fruits to accurately predict the number of red fruits. Results reveal that TomPhenoNet achieves high prediction performances, with R 2 values of 0.828, 0.930, 0.945, and 0.881 for plant height, leaf area, fresh weight, and the number of red fruits, respectively. Ablation experiments show that the cross-stitch network and fruit detection features improve the prediction performances of growth parameters, with TomPhenoNet combining both modules performing best. Feature importance analysis indicates the network model captures plant growth characteristics and corrects the impact of leaf occlusion from the top view. This study promotes accurate tomato monitoring and provides data support for optimizing cultivation strategies.

Plant phenotyping relevance

トマトの複数形質をRGB-D画像から推定するマルチモーダル・マルチタスク手法を開発し、性能評価とアブレーション実験を行っており、表現型取得・推定が研究の中心である。

Code and data availability

公開論文であることは確認できましたが、現在の公式API・許可済み取得経路では本文を自動取得できませんでした。

No evidence-backed public reproduction asset is currently recorded.