We present a novel framework for landmark detection and tracking in ultrasound data. Our method employs a convolutional neural network (CNN) encoder-decoder for landmark detection, coupled with a recurrent neural network (RNN) for encoding information from previous video frames of the object being tracked. We evaluated our method on the MICCAI CLUST 2015 De Luca u.a. (2015) challenge dataset, and have achieved promising results.