The present invention discloses a video summarization system and the method thereof. A similarity computing apparatus computes the similarity between each frame to obtain multiple similarity values. A key frame extracting apparatus chooses the key frames from the frames wherein the sum of the similarity values between the key frames is a minimum. A feature space mapping apparatus converts the sentences into multiple corresponding sentence vectors and computes the distance between each sentence vector to obtain multiple distance values. A clustering apparatus divides the sentences into multiple clusters according to the distance values and the importance of the sentences, and also applies a splitting step to split the cluster with the highest importance into multiple new clusters. A key sentence extracting apparatus chooses multiple key sentence from the clusters, wherein the sum of the importance of the key sentences is the maximum.