Semantic segmentation model of multi-source remote sensing images was used to extract winter wheat at tillering stage.

Abstract

In complex farmland environments, wheat canopy coverage is insufficient at the tillering stage, posing a considerable challenge to the accurate extraction of its canopy using UAV(unmanned air vehicle) remote sensing images. In this paper, an end-to-end semantic segmentation method based on visible light (RGB) and thermal infrared (TIR) images, Tiff-SegFormer, which fused spectral features and temperature features effectively, is proposed for accurate pixel-level classification of winter wheat at the tillering stage images taken by UAV to segment the wheat canopy and background. Tiff-SegFormer utilizes hierarchical feature representation and efficient self-attention in the encoder stage to extract features of detail contours of RGB images and temperature changes of TIFF images, respectively. In the decoder stage, the features are concatenated and then the channel and spatial attention mechanisms are superimposed, aiming to further improve the segmentation accuracy and efficiency of winter wheat at the tillering stage in UAV remote sensing images. The results show that Tiff-SegFormer can achieve accurate segmentation of wheat canopy and background from UAV images of winter wheat at the tillering stage (mIoU = 84.28%, mPA = 88.97%, accuracy = 94.55%). In order to verify the efficiency of the proposed method, Tiff-SegFormer is compared with four widely used semantic segmentation methods, all of which show better performance. The four methods are UNet, DeepLabv3+, HRNet, SegFormer and four-channel (RGB + TIFF) Segformer. The generalization test shows that the proposed Tiff-SegFormer also achieves better performance than other comparison methods (mIoU = 84.94%, mPA = 91.46%, accuracy = 94.71%). Tiff-SegFormer provides a robust and efficient tool for segmenting winter wheat canopy from UAV remote sensing images of winter wheat at the tillering stage, and has great potential in applications (model implementation and results can be found at https://github.com/wylSUGAR/Tiff-SegFormer ).

Plant phenotyping relevance

UAVのRGB・熱赤外画像から冬コムギのキャノピーを抽出するセマンティックセグメンテーション手法を開発し、複数手法との比較および汎化性能検証を行っており、植物状態の取得方法が中心である。

Code and data availability

The paper publicly releases the UAV RGB/TIR winter wheat tillering-stage image dataset and the TIR-to-TIFF conversion code via two author GitHub repositories. The Tiff-SegFormer model repository is referenced but its URL is not among the allowed URLs, and labelme is a generic third-party tool, so neither is included.