A damper adjusting device is mounted on a base of an exercise apparatus, including a resistance wheel, a pivoting member, a driving mechanism, and a biasing member. The resistance wheel is rotatably mounted on the base. The pivoting member is pivotably mounted on the base, having a driven portion and a resistance portion. The driving mechanism is mounted on the base, having a rotatable turntable and a lug formed on the turntable and spaced from a center of the turntable. The biasing member generates resilience for keeping the driven portion contacting against the lug. Accordingly, when the lug is driven to move to further drive the pivoting member to pivot for an angle, the distance between the resistance portion and the resistance wheel is also changed to shift the resistance generated while the resistance wheel turns.