An automated face enhancement system and process which can automatically improve faces in videos or other images by applying cosmetic effects, given only a small amount of user interaction for initialization. For an input video, the system will first track the face and eye locations, and classify pixels in the face into different facial components based on local color models. The classification results of different frames are temporally smoothed to ensure temporal coherence. A set of cosmetic filters are then applied to different facial components.