Temporal semantic multispectral point cloud generation and feature fusion pipeline for comprehensive trait estimation in greenhouse tomatoes

Abstract

Accurate estimation of comprehensive traits such as yield and quality is crucial for optimizing agricultural management practices across the tomato industry chain. Traditional manual methods are time-consuming, labor-intensive, and prone to errors, reducing estimation accuracy. In contrast, modern intelligent estimation approaches based on multi-temporal spatial and spectral feature fusion offer improved efficiency and accuracy but still face challenges such as non-generalizable segmentation models, asynchronous feature extraction and weak correlations. This study proposes a novel pipeline for estimating yield and quality of greenhouse tomatoes using temporal semantic multispectral (TSM) point clouds. An unsupervised deep learning model was designed to register RGB-D images and multispectral (MS) images collected by an unmanned ground vehicle (UGV) plant phenotyping platform. The digital number (DN) point clouds of tomato organs were reconstructed based on the masks predicted by SegFormer with fusion of multispectral and depth modalities (MSD-SF). These point clouds were then radiometrically calibrated using neural reference field with sparse viewpoints (NeREF-S) to generate accurate reflectance point clouds. Finally, multi-temporal spatial-spectral features of tomatoes were extracted from the TSM point clouds, and random forest regression models were developed to estimate traits such as fruit flavor preference, water content, brix, acidity, brix-to-acid ratio, vitamin C content, single-fruit mass, and single-plant yield. The image registration model achieved high accuracy on the test set, with average structural similarity index measure, peak signal-to-noise ratio and learned perceptual image patch similarity of 0.238, 13.116 dB, and 0.374, respectively. The MS point clouds calibrated by NeREF-S significantly improved the signal-to-noise ratio to 11.56 dB. The average rRMSE for all trait estimations was 9.03 %. The results indicate that the proposed estimation method is efficient and accurate, holding promise to become a new paradigm for estimating the comprehensive traits of greenhouse tomatoes.

Plant phenotyping relevance

温室トマトの収量・品質形質を推定するため、UGVフェノタイピングプラットフォーム、マルチスペクトル点群生成、画像登録・放射較正、特徴抽出および回帰推定パイプラインを中心的に開発・評価している。

Code and data availability

公開状態または取得可能な本文経路を確認できませんでした。

No evidence-backed public reproduction asset is currently recorded.