Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion

Abstract

We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

Plant phenotyping relevance

果実数を画像から抽出する深層学習・追跡・SfM統合パイプラインを開発し、アノテーション済み計数で精度検証しており、植物表現型取得が中心である。

Code and data availability

The paper describes fruit counting on orange and apple image datasets, but no public dataset, code, model, or data deposit is stated. The only public links are an algorithm demonstration video (YouTube / label.ag), which is a result demo rather than a phenotype dataset, sensor inputs, analysis code, or trained model,so

No evidence-backed public reproduction asset is currently recorded.