A collet system for a rotary tool includes a collet for receiving a bit; a wrench assembly for engaging the collet, the wrench assembly moving along an axis extending axially through the collet between a first position where the wrench assembly is disengaged from the collet and a second position where the wrench assembly is engaged with the collet; and an adjusting collar coupled to the wrench assembly for moving the wrench assembly between the first position and the second position, wherein upon moving the wrench assembly to the second position, the adjusting collar is rotatable for rotating the wrench assembly to tighten and loosen the collet.