A system and method for tracking a global shape of an object in motion is disclosed. One or more control points along an initial contour of the global shape are defined. Each of the one or more control points is tracked as the object is in motion. Uncertainty of a location of a control point in motion is represented using a number of techniques. The uncertainty to constrain the global shape is exploited using a prior shape model. In an alternative embodiment, multiple appearance models are built for each control point and the motion vectors produced by each model are combined in order to track the shape of the object. The movement of the shape of the object can be visually tracked using a display and color vectors.