Hybrid deep learning-based multimodal framework for plant leaf disease classification using RGB, Excess Green (ExG), and pseudo-thermal representations with MobileNetV2

Abstract

Abstract Plant diseases are a serious danger to the world’s food security, because they lower agricultural output and increase economic losses. Due to subjectivity, fluctuating lighting, and environmental unpredictability, traditional visual examination techniques are frequently incorrect. The Excess Green (ExG) vegetation index and pseudo-thermal representations produced from RGB pictures are two synthetically developed complementary representations that are integrated with RGB imagery in this study’s lightweight multimodal deep learning system to address these issues. Histogram shifting and pseudo-infrared color mapping are used in a reproducible picture alteration pipeline to create the pseudo-thermal modality, which allows for extra visual signals without the need for specific thermal sensors. In order to classify plant diseases while preserving computational efficiency, the suggested framework uses MobileNetV3-Small backbones to extract modality-specific characteristics. This is followed by feature-level fusion. The publicly accessible Ginger Leaf Dataset, which includes RGB pictures of ginger leaves in four different conditions—Damage-Pest, Dehydrated, Healthy, and Leaf-blight—was used for the experiments. For training, validation, and testing, the dataset was split using a stratified 70:15:15 split. Python-based preprocessing procedures were used to create the extra modalities (ExG and pseudo-thermal representations) from the original RGB images. The experimental results show that the combination of the representations with RGB images can enhance the classification performance compared with the unimodal RGB-based models. Ablation experiments are also conducted to examine the contributions of different modalities to the overall categorization accuracy. The experimental results show that plant disease recognition can be improved with the help of efficient computing by combining lightweight convolutional neural networks with computationally generated visual representations.

Plant phenotyping relevance

RGB画像からExG・疑似熱画像を生成し、植物葉の病害状態を分類するマルチモーダル手法が研究の中心であり、アブレーション評価も実施している。

Code and data availability

The paper's phenotyping experiments use the publicly available Ginger Leaf Dataset (RGB leaf images of four ginger leaf conditions), with a public GitHub repository and dataset website. The authors' derived ExG/pseudo-thermal representations and preprocessing scripts are only available upon request, so they do not yet