An interior rearview mirror system for a vehicle includes an interior rearview mirror assembly, a casing having a reflective element, a compass sensor and a control. The compass sensor is disposed within the mirror casing and has a first magnetoresponsive sensing element and a second magnetoresponsive sensing element. The control receives a first signal indicative of a magnetic field sensed by the first magnetoresponsive sensing element and a second signal indicative of a magnetic field sensed by the second magnetoresponsive sensing element. The control determines that the casing is adjusted by an occupant of the vehicle in response to a change in the first and second signals being indicative of an abrupt movement of the casing about the mounting structure by an occupant of the vehicle. The control is operable to enter a rapid compensating mode to compensate for the mirror adjustment.