From blender to farm: Transforming controlled environment agriculture with synthetic data and SwinUNet for precision crop monitoring

Abstract

The aim of this study was to train a Vision Transformer (ViT) model for semantic segmentation to differentiate between ripe and unripe strawberries using synthetic data to avoid challenges with conventional data collection methods. The solution used Blender to generate synthetic strawberry images along with their corresponding masks for precise segmentation. Subsequently, the synthetic images were used to train and evaluate the SwinUNet as a segmentation method, and Deep Domain Confusion was utilized for domain adaptation. The trained model was then tested on real images from the Strawberry Digital Images dataset. The performance on the real data achieved a Dice Similarity Coefficient of 94.8% for ripe strawberries and 94% for unripe strawberries, highlighting its effectiveness for applications such as fruit ripeness detection. Additionally, the results show that increasing the volume and diversity of the training data can significantly enhance the segmentation accuracy of each class. This approach demonstrates how synthetic datasets can be employed as a cost-effective and efficient solution for overcoming data scarcity in agricultural applications.

Plant phenotyping relevance

合成画像、セグメンテーション、ドメイン適応を用いてイチゴの成熟状態を推定する画像解析手法を開発・実データで評価しており、植物器官の状態取得が中心である。

Code and data availability

The authors state that all data (synthetic strawberry images and masks) and all Python analysis code are publicly available on the Open Science Framework at https://osf.io/5kzcb/, making both the paper-specific phenotype/segmentation dataset and the authors' code directly actionable.