Deep spatial-spectral fusion of UAV RGB and hyperspectral imagery for potato plant disease detection

Abstract

Accurate plant disease detection remains challenging when using single-modality data, which fails to capture comprehensive disease-related features. However, many existing studies rely on pixel-level classification or prior plant segmentation and lack explicit modeling of cross-modal interactions, limiting their ability to distinguish between healthy and diseased plants. This study presents a spatial–spectral fusion deep learning framework (S2-PDD) that fuses uncrewed aerial vehicles (UAV)-based RGB imagery and hyperspectral-derived feature representations for plant-level detection of potato diseases (blackleg and Potato virus Y (PVY)), employing early fusion (modalities combined at the input stage) and middle fusion (features integrated at intermediate stages within the model backbone) strategies. The multimodal fusion models were compared against single-modal models and an existing S2ADet model. Model performance, assessed through five-fold cross-validation, demonstrated that multimodal models integrating RGB and vegetation index features achieved the highest mAPs of 86.65 ± 1.71 (%; E-RV model) and 85.74 ± 1.96 (M-RV), respectively. These mAPs were higher than those of all single-modal models, including the RGB-only (83.21 ± 1.46) and hyperspectral-only models (PCA features: 79.71 ± 1.45; vegetation index features: 85.31 ± 2.36). They also exceeded mAPs of multimodal models combining RGB with PCA features (early fusion: 83.00 ± 2.81; middle fusion: 83.11 ± 2.46; S2ADet: 84.04 ± 2.73), regardless of the fusion strategy. The superior performance highlights that vegetation index features provide strong class separability compared to other hyperspectral representations. The proposed models achieved strong plant-level detection performance, with AP of 78.65 ± 4.14 (E-RV) and 77.40 ± 3.46 (M-RV) for blackleg disease, as well as 84.82 ± 3.59 (E-RV) and 83.28 ± 3.86 (M-RV) for PVY. These results demonstrate the potential of UAV-based multimodal sensing for disease monitoring in cropping systems. A potato plant disease detection dataset was constructed and made publicly available, containing paired RGB and hyperspectral image tiles with bounding box annotations. The code is available at https://github.com/Tim-Agro/S2-PDD.

Plant phenotyping relevance

UAV RGB・ハイパースペクトル画像からジャガイモ個体の病害状態を推定する融合モデルを開発・比較検証し、公開データセットも構築しており、病害表現型の取得・抽出手法が中心である。

Code and data availability

公開論文であることは確認できましたが、現在の公式API・許可済み取得経路では本文を自動取得できませんでした。

No evidence-backed public reproduction asset is currently recorded.