An image data compression device includes: a quantization section which quantizes image data in unit of a frame; a FIFO buffer section in which quantized data for at least N frames (N is an integer larger than one) quantized by the quantization section is buffered; an encoded data generation section which reads the quantized data from the FIFO buffer section asynchronously from writing into the FIFO buffer section, and generates encoded data; and a rate control section which changes a data size of the encoded data by changing a quantization step of the quantization section by using the data size of the encoded data. The rate control section obtains an average data size by averaging the data sizes of the encoded data for N frames before a frame of the image data quantized by the quantization section, and changes the quantization step based on the average data size.