PhenoNEST: A Neuro-Symbolic Framework for Ontology-Aware Multimodal Plant Phenotyping and Trait Discovery

Abstract

High-throughput plant phenotyping generates valuable data that often remains trapped in unstructured text and isolated RGB images. To bridge this semantic gap, we propose a framework for constructing a multimodal granular Knowledge Graph (KG) to monitor genotype-phenotype interactions across time and experiments. In this work, we focus on wheat Triticum aestivum as a representative target crop to validate our methodology across complex canopy environments. Our pipeline first distills noisy field notes to extract entities and relations, dynamically constructing the KG by converting unique instances into hierarchical class entities via RDF-typing. These graph nodes are then aligned with standardized ontologies (PO, RO, WTO) using PlantDeBERTa. To visually ground the constructed graph, a Vision-Language Model paired with a wheat-segmentation ViT generates attention-based softmaps, linking specific KG entities directly to image pixels. We introduce a central observation node Plant_Obs_Id to connect these multimodal subgraphs temporally. Evaluated on 500 curated WisWheat samples using Pointing Game accuracy, Visual Word Sense Disambiguation (VWSD), and rank-based metrics, our neuro-symbolic approach successfully maps complex field observations to a structured graph. This enables automated field note auditing, temporal stress monitoring, and precise spatial trait localization for wheat breeders.

Plant phenotyping relevance

植物のマルチモーダル表現型データを知識グラフと画像に統合し、画像画素への形質局在化を行う中核的な計算フレームワークを提案・評価しているため。

Code and data availability

The paper's own analysis code (PhenoNEST framework) is not publicly deposited; authors state it will be shared only upon reasonable request. All datasets used (WisWheat, GWHD, GWFSS, WFD2020, TaeC) are cited prior public resources, not paper-specific assets, and PlantDeBERTa is an external cited model.

No evidence-backed public reproduction asset is currently recorded.