Digital Phenotyping in Plant Breeding: Evaluating Relative Maturity, Stand Count, and Plant Height in Dry Beans (Phaseolus vulgaris L.) via RGB Drone-Based Imagery and Deep Learning Approaches

Abstract

Abstract Background Significant effort has been made in manually tracking plant maturity and to measure early-stage plant density, and crop height in experimental breeding plots. Agronomic traits such as relative maturity (RM), stand count (SC) and plant height (PH) are essential to cultivar development, production recommendations and management practices. The use of RGB images collected via drones may replace traditional measurements in field trials with improved throughput, accuracy, and reduced cost. Recent advances in deep learning (DL) approaches have enabled the development of automated high-throughput phenotyping (HTP) systems that can quickly and accurately measure target traits using low-cost RGB drones. In this study, a time series of drone images was employed to estimate dry bean relative maturity (RM) using a hybrid model combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) for features extraction and capturing the sequential behavior of time series data. The performance of the Faster-RCNN object detection algorithm was also examined for stand count (SC) assessment during the early growth stages of dry beans. Various factors, such as flight frequencies, image resolution, and data augmentation, along with pseudo-labeling techniques, were investigated to enhance the performance and accuracy of DL models. Traditional methods involving pre-processing of images were also compared to the DL models employed in this study. Moreover, plant architecture was analyzed to extract plant height (PH) using digital surface model (DSM) and point cloud (PC) data sources. Results The CNN-LSTM model demonstrated high performance in predicting the RM of plots across diverse environments and flight datasets, regardless of image size or flight frequency. The DL model consistently outperformed the pre-processing images approach using traditional analysis (LOESS and SEG models), particularly when comparing errors using mean absolute error (MAE), providing less than two days of error in prediction across all environments. When growing degree days (GDD) data was incorporated into the CNN-LSTM model, the performance improved in certain environments, especially under unfavorable environmental conditions or weather stress. However, in other environments, the CNN-LSTM model performed similarly to or slightly better than the CNN-LSTM + GDD model. Consequently, incorporating GDD may not be necessary unless weather conditions are extreme. The Faster R-CNN model employed in this study was successful in accurately identifying bean plants at early growth stages, with correlations between the predicted SC and ground truth (GT) measurements of 0.8. The model performed consistently across various flight altitudes, and its accuracy was better compared to traditional segmentation methods using pre-processing images in OpenCV and the watershed algorithm. An appropriate growth stage should be carefully targeted for optimal results, as well as precise boundary box annotations. On average, the PC data source marginally outperformed the CSM/DSM data to estimating PH, with average correlation results of 0.55 for PC and 0.52 for CSM/DSM. The choice between them may depend on the specific environment and flight conditions, as the PH performance estimation is similar in the analyzed scenarios. However, the ground and vegetation elevation estimates can be optimized by deploying different thresholds and metrics to classify the data and perform the height extraction, respectively. Conclusions The results demonstrate that the CNN-LSTM and Faster R-CNN deep learning models outperforms other state-of-the-art techniques to quantify, respectively, RM and SC. The subtraction method proposed for estimating PH in the absence of accurate ground elevation data yielded results comparable to the difference-based method. In addition, open-source software developed to conduct the PH and RM analyses can contribute greatly to the phenotyping community.

Plant phenotyping relevance

RGBドローン画像と深層学習を用いて、乾燥豆の成熟期、株数、草高を推定する手法を開発・比較・検証しており、表現型取得が研究の中心である。

Code and data availability

The preprint explicitly states that the authors' open-source phenotyping software (RM, SC, PH pipelines) is available on GitHub, with specific tools (matuRity, Vegetation index calculator, PlantHeightR, draw-plots-qgis) hosted at public URLs, and that the datasets (orthomosaics, shapefiles, ground notes, clipped plots,