A method and system for estimating motion between a reference image and a search image of a video image sequence. Motion vectors are determined for pixel blocks of the reference image of the video image sequence. The major modes of the motion vectors for the reference image are identified and used for determining each pixel block of the reference image, if a better match can be made based on the major modes of the motion vectors.