A system and method for classifying follicular units based on the number of hairs in a follicular unit of interest comprises acquiring an image of a body surface having a follicular unit of interest, processing the image to calculate a contour of the follicular unit and an outline profile which disregards concavities in the contour, and determining the number of defects in the outline profile to determine the number of hairs in the follicular unit. The system and method may also adjust for hairs which converge beneath the skin and for images which appear as a single wide hair but which are actually multiple hairs. In another aspect, a system and method for determining the end points of a follicular unit comprises generating a skeleton of a segmented image and identifying the end points from the skeletonized image.