A video skim is assembled by identifying one or more key frames from a video shot. Certain lengths of frames to the left and right of the key frame are measured for visual content variety. Depending upon the measured visual content variety to the left and right of the key frame, the video skim is assembled that has L frames to the left of the key frame and R frames to the right of the key frame. Measuring the visual content variety to the left and right of the key frame, provides a video skim that incorporates the more salient features of a shot.