Fish Species Classification in Controlled Underwater Environments Using Contrastive Language-Image Models

Abstract

This thesis investigates the application of contrastive language-image pretraining models for automated fish species classification in controlled underwater monitoring environments. Using the SigLIP model architecture, a comprehensive pipeline was developed to detect and classify three species of salmonids (Salmon, Trout, and Char) from video data collected at Icelandic river monitoring stations.The methodology leverages zero-shot learning capabilities for fish detection and multiple fish identification, followed by temporal feature extraction for species classification. Key innovations include prompt engineering optimization for detection tasks and temporal feature aggregation strategies that improve classification performance while maintaining computational efficiency.The results demonstrate strong performance across all tasks: fish detection achieved 99.1% accuracy with perfect precision, multiple fish detection reached 98.2% accuracy, and species classification achieved a macro F1-score of 97.8% using SVM with temporal pooling. These results were achieved with significantly less training data compared to methods without temporal aggregation and other traditional deep learning approaches, while maintaining robust performance under varying underwater conditions.The primary contribution of the study is to demonstrate that embedding models can effectively handle both detection and classification tasks in controlled underwater environments without frame-level annotations or extensive preprocessing. However, limitations include the dependency on high-quality video segments and potential constraints on generalization to unconstrained environments. These findings have implications for developing more efficient and maintainable automated monitoring systems for marine biology and ecological research.Keywords: fish species classification, zero-shot learning, contrastive learning, underwater monitoring, temporal feature extraction, computer vision