A video endoscope apparatus includes a light source unit that alternately emits white light and excitation light, a video endoscope that has a color image sensor and a light guide for guiding the light from the light source unit to the distal end, an image data acquiring section that acquires normal observation image data during emission of the white light and acquires excitation observation image data during emission of the excitation light, first and second image processors, an image generating section. The first image processor processes the normal observation image data only. The second image processor extracts reference image data from the normal observation image data, extracts fluorescent image data from the excitation observation image data and generates affected part information. The image generating section generates image data for displaying a special observation image by converting the normal observation image data based on the affected part information.