RT-ZSDR: A real-time zero-shot segmentation and dense reconstruction framework for plant phenotyping robot

Abstract

Annotation scarcity, poor model generalization and lagged data processing remain key bottlenecks hindering the practical deployment of phenotyping robots. To address these issues, we developed a novel phenotyping robot capable of online 3D reconstruction and zero-shot segmentation directly on the edge. Diverging from conventional semantic SLAM, our core contribution is RT-ZSDR, a framework featuring two key methodological novelties. First, we introduce the ForeCut pipeline for target extraction, which innovatively fuses DINO features with 3D geometric spatial information, leveraging multi-view semantic-spatial consistency to achieve annotation-free, zero-shot dense segmentation and reconstruction. Second, we designed a hardware-coupled loop closure strategy utilizing the robotic arm's kinematic feedback as prior constraints to significantly improve loop closure recall. Supported by edge computing Jetson Orin NX, the tracking and segmentation process takes approximately 0.24 s per frame after an initialization period of 1.82 s. RT-ZSDR's phenotypic measurements demonstrated strong correlations with reference baseline in both laboratory settings (n=90, PlantEye measurements as reference baseline; R 2 =0.990, 0.939, 0.725, and 0.861 for plant height, projected leaf area, surface area, and volume) and practical greenhouse environments (n=48, manual measurements as reference baseline; R 2 =0.965, 0.862 for plant height and stem diameter). Additionally, evaluated against COLMAP benchmarks (n=24), the system achieved a mean 3D reconstruction F1-score of 0.816.

Plant phenotyping relevance

植物フェノタイピングロボット向けに、ゼロショット分割・3D再構成・エッジ処理を開発し、植物形質を基準測定およびベンチマークと比較検証しており、取得・抽出手法が研究の中心である。

Code and data availability

公開論文であることは確認できましたが、現在の公式API・許可済み取得経路では本文を自動取得できませんでした。

No evidence-backed public reproduction asset is currently recorded.