An image processing apparatus includes a classification unit configured to extract N features from an input image using pre-generated N feature extraction units and calculate confidence value which represents object-likelihood based on the extracted N features, an object detection unit configured to detect an object included in the input image based on the confidence value, a feature selection unit configured to select M feature extraction units from the N feature extraction units such that separability between the confidence value of the object and that of background thereof becomes greater than a case where the N feature extraction units are used, the M being a positive integer smaller than N, and an object tracking unit configured to extract M features from the input image and tracks the object using the M features selected by the feature selection unit.