The semi-fragile watermark comprises a fragile watermark component and a robust watermark component. Features are extracted from the video stream and subsequent hashing and encryption processes are performed to generate the fragile watermark. The fragile watermark includes control data information at the block level as well as frame and group level information. The fragile watermark is added on top of the robust watermark, giving the system the ability to detect alteration at the block level as well as the group level. The resulting semi-fragile watermark has the advantage of being both sensitive to malicious attack while being robust enough to survive bit rate reduction and other types of manipulation typically performed on digital multimedia signals.