A method of inspecting pattern defects can detect target defects in various processes stably by reducing erroneous detection of grains and morphology and decreasing the influence of an intensity nonuniformity in interference light. For this purpose, lights emitted from two sources of illumination capable of outputting a plurality of wavelengths are reflected by a beam splitter and irradiated onto a wafer. Diffracted light from the wafer is converged by an objective lens, is made to pass through light modulation units and imaged on an image sensor in a light detection unit. Then, defects are detected in a signal processing unit. Further, the optical modulation unit is made to have a structure that uses a plurality of optical components selectively and which can be optimized according to target defects.