An image data compression device includes a quantization section, a FIFO buffer section, an encoded data generation section, and a rate control section. The quantization section quantizes image data by a quantization step which changes based on a quantization parameter. The FIFO buffer section stores the quantized data for at least N frames (N is an integer greater than one) from the quantization section. The encoded data generation section generates encoded data by encoding the quantized data read asynchronously from writing into the FIFO buffer section. The rate control section calculates the quantization parameter by using an average data size of encoded data for previous N frames. The quantization parameter is calculated to be equal to or less than an upper threshold of the quantization parameter which is set or canceled based on the number of assignable bits which can be assigned to the compression processing and the number of remaining frames.