A brightness adjusting method for video frames of a video sequence includes receiving a current video frame of the video sequence and performing a scene change detection to determine if a scene change occurs between the current video frame and a previous video frame. Thereafter, one of a plurality of candidate brightness adjusting schemes is applied to the current video frame according to a scene change detection result.