A human eye is detected in a video image by identifying potential eye candidates using horizontal and vertical projection filtering on a specified portion of the image, coupled with rudimentary appearance-based testing. The identified eye candidates are clustered and winning eye candidates are selected based on appearance, spatial-geometric and anthropomorphic considerations. The winning eye candidates are subjected to additional appearance-based testing, and high confidence eye candidates are accepted if they favorably compare with identified eye candidates from previous video image frames.