A rotary tool holder includes a shank having a flange, a tapered outer surface, and front and rear contact portions. The tapered outer surface corresponds to the tapered bore of a spindle, and flexible circular cantilevers are provided at one or both contact portions such that a free end of the cantilevers expands radially due to centrifugal force to maintain contact with the tapered bore. The shank can be formed from an inner member and a sleeve disposed thereover, in which portions of the sleeve form the circular cantilevers. The circular cantilevers can also be formed by a cavity created in the shank. Alternatively, instead of circular cantilevers, the shank of the tool holder can have a taper which is optimized for a certain speed, wherein the taper of the shank has a first taper at rest and a different, optimized taper at a desired speed.