Multi-modal deep learning for paddy health assessment: fusing leaf imagery with tabular metadata using a factorized bilinear pooling approach.

Abstract

Global food security is largely based on the accurate and timely diagnosis of crop diseases, where paddy rice is an extremely essential staple of more than half of the world population. The conventional disease identification techniques tend to be laborious, time consuming and demand a great deal of domain knowledge, which becomes a bottleneck in the efficient management of the farms. Although deep learning [and especially Convolutional Neural Networks (CNNs)] have demonstrated a spectacular performance in automated classification of diseases based on leaf images, they tend to overlook important contextual features that are implicitly processed by agronomic experts. The visual defects of a disease might be unclear and this can greatly differ depending on factors like the genetic variety of the plant and the stage of development. We overcome this shortcoming by proposing a new multi-modal deep learning framework, Multi-Modal Factorized Bilinear Pooling (MFBP) model which is capable of a more holistic and precise paddy health measurement. The proposed method is the only one that combines high-level visual information obtained using leaf images and related tabular information, namely the paddy type and number of days. The MFBP model uses Factorized Bilinear Pooling (FBP) rather than the simple feature concatenation which commonly loses the complex relationship between different data types. This systematic method efficiently encodes all the complex interactions between all components of the visual and tabular features vectors in such a way that helps the model to pick up subtle, context-specific patterns. As an example, it will only be possible to educate the model that a specific visual blemish is predictive of a given disease through a specific species at a specific age. We test our model on the Paddy Doctor: Paddy Disease Classification dataset, which is a detailed public dataset comprising of more than 10,000 labeled images and containing relevant metadata, and thus it forms a perfect testing bed to conduct multi-modal research. Through our detailed experiments, we have shown that the proposed MFBP model is much better than a baseline model based on concatenation fusion, which proves that deep, multiplicative interactions can be best modeled in this task. The findings highlight the massive possibilities of multi-modes AI in the development of more robust, more accurate, and more context-aware diagnostic instruments and precision agriculture to enable more sustainable and productive agricultural activities.

Plant phenotyping relevance

葉画像とメタデータを統合してイネの健康状態・病害を推定する新規深層学習手法を提案し、ベースライン比較で検証しているため、植物フェノタイピング手法が中心である。

Code and data availability

The paper's phenotyping inputs are the public Kaggle 'Paddy Doctor: Paddy Disease Classification' dataset (10,407 leaf images with tabular metadata for variety and age), explicitly named in the Data Availability statement with a persistent public URL. No author analysis code, trained models, or checkpoints are reported