Enhanced machine vision system for field-based detection of pickable strawberries: Integrating an advanced two-step deep learning model merging improved YOLOv8 and YOLOv5-cls

Abstract

In order to successfully deploy robotic harvesting in open field conditions, the development of an effective machine vision system becomes crucial. In this research, we proposed a novel two-step deep learning model consisting of a modified YOLOv8s and a YOLOv5s-cls to accomplish strawberry detection and pickability classification (whether a mature fruit is pickable by a robot). Firstly, the YOLOv8s was enhanced by incorporating C3x modules and an additional head network structure, specifically tailored for accurate strawberry detection. To further improve training performance, the α-IOU (intersection over union) technique was integrated. Subsequently, the YOLOv5s-cls was utilized to determine suitability of the detected mature strawberries. Through evaluations, Model D (+C3x+head+αIoU), which was a model based on modifying YOLOv8 using the new modules and techniques mentioned above, was found to perform the best among the tested models achieving the highest AP scores of 84.2% in Stage I (immature), 77.8% in Stage II (nearly mature), and 87.8% in Stage III (mature), along with the highest mAP of 83.2%. Overall, this modified model achieved a 2.5% improvement in mAP compared to the same achieved by original YOLOv8s model. Despite a slightly slower inference speed of 8.4 ms per image, Model D maintains real-time capabilities, making it an optimal choice for strawberry detection. Additionally, YOLOv5s-cls was identified as the preferred model for classifying mature strawberries into pickable and unpickable groups, offering a good inference speed of 2.8 ms per image and comparable accuracy with other compared models including YOLOv8s-cls, ResNet 18, EfficientNet-b0, and EfficientNet-b1. Finally, the combined two-step model developed in this study was evaluated in 10 different field scenarios from a completely different strawberry field that was not used in model training and initial testing. In this validation test the machine vision system achieved an AP of 89.0%, 82.0%, and 90.0% in detecting strawberries from Stage I, II, and III while the classification accuracy was 100.0% in unpickable group and 95.0% in pickable group. The results showed that the developed two-step machine vision system has a potential to improve the overall robotic harvesting system for strawberries grown in open-field conditions.

Plant phenotyping relevance

イチゴ果実の成熟段階と収穫可能性という植物器官の状態を推定する機械視覚法を開発し、別圃場で検証しており、単なる収穫対象の位置検出を超えた中心的なフェノタイピング手法である。

Code and data availability

公開状態または取得可能な本文経路を確認できませんでした。

No evidence-backed public reproduction asset is currently recorded.