Species-level detection and tracking of fish from videos is a crucial capability for ecologists to study and understand trends in bio-diversity as they relate to ecosystem changes. However, these capabilities still trail their terrestrial equivalents. This paper presents a dataset and initial experiments to understand the current state and limitations for this technology, highlighting the constraints encountered in an ecological fieldwork scenario. Our dataset focuses on Caribbean coral reef fish, where ecologists have gathered consistent data over 8 years across 5 distinct reef sites. This dataset leverages diver-collected visual survey data, unlike alternative datasets, which typically either rely on stationary, baited camera traps or alternative methods of sourcing data, such as caught animals or tourist captured videos. Our dataset consists of 14,580 expert-annotated frames, including 15,802 unique fish tracks consisting of 98,267 bounding boxes. Of those tracks, 2,410 of them include species-level annotations from 54 species found in these waters. Our preliminary experiments highlight the challenges in applying typical fish datasets and detection methods to a novel ecological context.