An illumination device to increase or decrease a quantity of light, while maintaining a uniform illumination intensity distribution, an image display device that displays high-quality images with a wide dynamic range, and a projector are provided. The illumination device includes a light source including a lamp and a concave mirror and emitting an illumination light, a variable diaphragm, for example in the form of a shutter having a linear edge and controlling the quantity of the illumination light, and a rotary prism which is an illumination light scanning device to scan the illumination light that passed through the variable diaphragm over an irradiated surface. The axis of the direction in which the illumination light is scanned over the irradiated surface almost coincides with the axis of the direction in which the variable diaphragm controls the illumination light. Here, the control modes in the +x axis direction, -x axis direction and .+-.x axis direction are used to match the axis.