An alpha-map encoding apparatus includes a first down-sampling circuit (21) for down-sampling an alpha-map signal which represents the shape of an object and the position in the frame of the object at a down-sampling ratio based on size conversion ratio information, an up-sampling circuit (23) for up-sampling the alpha-map signal at an up-sampling ratio based on size conversion ratio information given to restore the down-sampled-alpha-map signal to an original size, and outputting a local decoded alpha-map signal, a motion estimation/compensation circuit (25) for generating a motion estimation/compensation signal on the basis of the previous decoded video signal and a motion vector signal, a second down-sampling circuit (26) for down-sampling the motion estimation/compensation signal at the down-sampling ratio, a binary image encoder for encoding the alpha-map signal down-sampled by the first down-sampling circuit to a binary image in accordance with the motion estimation/compensation signal down-sampled by the second down-sampling circuit, and outputting an encoded binary image signal, and a multiplexer for multiplexing and outputting the encoded binary image signal and the up-sampling ratio information.