A method for detecting human skin tone in a video signal by comparing image data from the video signal to a color profile table obtained from a plurality of training sequences. The image data comprising a plurality of pixels each having a plurality of color components, the components preferably being in the YCbCr color space. The method includes separating image data for each frame in the video signal into sets of data, averaging the image data in each data set to generate mean values for each color component in the data set, comparing the mean values to a stored color profile correlating to human skin tone, and identifying data sets falling within the stored color profile.