An apparatus and method of motion estimation and mode decision includes a candidate vector group generation unit generating a candidate vector group including a motion vector of an adjacent image crossing a first macro block, where the first macro block is a macro block of an image before scaling, and the first macro block and the adjacent image exist in a same sub-group of a picture (sub-GOP). A motion vector estimation unit estimates a base vector of a second macro block from the motion vector and the candidate vector group of the first macro block overlapped with the second macro block, where the second macro block is a macro block of an image after scaling. A mode decision unit determines a mode of the second macro block from the estimated base vector and a mode of the first macro block.