Systems and methods are described for tracking motion in a medical imaging system. The methods use landmark sets including hard landmarks and soft landmarks. The systems can include means for generating a landmark set, means for constructing a landmark set spatial relationship, means for computing a context similarity measurement, means for cost function construction, and means for cost function minimization and motion estimation. The invention provides advantages because, among other reasons, the use of a cost function of two or more variables, where the cost function is defined over members of the landmark set collection, and the cost function subject to at least two constraints and the landmark set stability in terms of context similarity measurements and spatial relationship provide effective and robust motion tracking.