A moving image Vd consists of a plurality of frames F.sub.1 F.sub.12. A scene change SC of the moving image Vd is detected according to change in each frame regarding to brightness information of image, and the plurality of frames are divided into frame groups FG1 (F.sub.1 F.sub.8) and FG2 (F.sub.9 F.sub.12) each having substantially the same scene at the scene change as a boundary. Then, with respect to the frame group FG1, every three frames, F.sub.1, F.sub.4, F.sub.7 are sampled to execute scene determination, and a most frequent scene is determined as a common scene of the frame group. Consequently, scene determination results for sufficiently scene-changed frames F.sub.4, F.sub.7 interposed between frames F.sub.1, F.sub.12 of both ends of the frame group FG1 can be utilized, so that it is possible to determine the common scene with high accuracy.