In-Vivo Biosensors and Visual Data for Precision Agriculture: a Multimodal Approach for Water Stress Detection in Tomato Plants

Early and accurate plant stress prediction is fundamental in precision agriculture to optimize resources use and crop yield. Herein, we introduce a multimodal framework for classifying water stress in tomato plants by exploiting data from novel in-vivo biosensors and plant images. We combine electronic features from the biosensors with RGB and NIR images captured from different points, exploiting a Transformer for the biosensor data and pretrained CLIP-based encoders for visual data, and we fuse them together before a cross-attention mechanism is applied. The system classifies plant health status into four health statuses. The results demonstrate better performance of multimodal model over single-modal baselines, and good results also in distinguishing ambiguous statuses. This demonstrates the effectiveness of the proposed multimodal framework for smart agriculture, with implications for sustainable crop management and water stress mitigation.