An apparatus and a method of encoding/decoding an input image frame in an image data processing system are disclosed. Using the apparatus and the method of the present invention, a total memory space of the memories required for encoding/decoding the input image frame is optimized using a slice memory. The encoder/decoder according to the present invention includes a slice memory storing each image slice of a restored image frame having T image slices and a first frame memory storing the restored image frame by copying each image slice stored in the slice memory until all of the T image slices are copied.