Framework for smartphone-based grape detection and vineyard management using UAV-trained AI.

Abstract

Viticulture benefits significantly from rapid grape bunch identification and counting, enhancing yield and quality. Recent technological and machine learning advancements, particularly in deep learning, have provided the tools necessary to create more efficient, automated processes that significantly reduce the time and effort required for these tasks. On one hand, drone, or Unmanned Aerial Vehicles (UAV) imagery combined with deep learning algorithms has revolutionised agriculture by automating plant health classification, disease identification, and fruit detection. However, these advancements often remain inaccessible to farmers due to their reliance on specialized hardware like ground robots or UAVs. On the other hand, most farmers have access to smartphones. This article proposes a novel approach combining UAVs and smartphone technologies. An AI-based framework is introduced, integrating a 5-stage AI pipeline combining object detection and pixel-level segmentation algorithms to automatically detect grape bunches in smartphone images of a commercial vineyard with vertical trellis training. By leveraging UAV-captured data for training, the proposed model not only accelerates the detection process but also enhances the accuracy and adaptability of grape bunch detection across different devices, surpassing the efficiency of traditional and purely UAV-based methods. To this end, using a dataset of UAV videos recorded during early growth stages in July (BBCH77-BBCH79), the X-Decoder segments vegetation in the front of the frames from their background and surroundings. X-Decoder is particularly advantageous because it can be seamlessly integrated into the AI pipeline without requiring changes to how data is captured, making it more versatile than other methods. Then, YOLO is trained using the videos and further applied to images taken by farmers with common smartphones (Xiaomi Poco X3 Pro and iPhone X). In addition, a web app was developed to connect the system with mobile technology easily. The proposed approach achieved a precision of 0.92 and recall of 0.735, with an F1 score of 0.82 and an Average Precision (AP) of 0.802 under different operation conditions, indicating high accuracy and reliability in detecting grape bunches. In addition, the AI-detected grape bunches were compared with the actual ground truth, achieving an R 2 value as high as 0.84, showing the robustness of the system. This study highlights the potential of using smartphone imaging and web applications together, making an effort to integrate these models into a real platform for farmers, offering a practical, affordable, accessible, and scalable solution. While smartphone-based image collection for model training is labour-intensive and costly, incorporating UAV data accelerates the process, facilitating the creation of models that generalise across diverse data sources and platforms. This blend of UAV efficiency and smartphone precision significantly cuts vineyard monitoring time and effort.

Plant phenotyping relevance

スマートフォン画像とUAVデータを用いてブドウ房を検出・計数するAIパイプラインを開発・評価し、実測値との比較も行っているため、植物器官形質の取得手法が中心である。

Code and data availability

The paper's Data Availability Statement points to a public, paper-specific dataset (EscaYard: geotagged smartphone vineyard images, phytosanitary status, UAV 3D point clouds and orthomosaics) published as a Data Brief with a DOI, which directly underpins the smartphone/UAV grape detection phenotyping analysis. No code,