A method of visual tracking with use of color includes the step of three-dimensional color modeling of the target, the step of target recognition for initially perceiving the target, and the step of visual tracking for repetitive visual tracking. In order to be robust to irregular or abrupt changes of illumination, the present invention provides a visual tracking method in which characteristics of the photographing element (CCD or CMOS) used by the camera are analyzed beforehand and the results of the analysis are modeled by a B-spline curve, allowing effects of the real-time visual tracking and adaptability to situations of rapid target movement.