According to the invention, channel surfers are aided by summary frames appearing on the screen along with the regular program. These summary frames are embedded in the broadcast and appear in a small window on the screen at some point in the future as the video is broadcast. The summary frames depict key scenes from the past which aid the viewer in quickly ascertaining the current plot or theme of the video program. In an interactive television environment, a short video segment may be associated with each shot such that when the shot is selected, the past video segment is played in a larger window on the screen complete with sound. Once the segment is viewed, the video program is rejoined in progress with the viewer having a better sense of what is going on in the video. Hence, a when a channel surfer arrives at a new channel, rather than only having what is currently playing to catch the eye, summary frames are also available to catch the surfer's attention and aid in understanding the programming.