A method and apparatus is disclosed for enhancing the efficiency of spatial scalable compression schemes by lowering the bitrate of the enhancement layer. The complete image is used during motion estimation and/or motion compensation in the enhancement layer by inserting both the reconstructed video stream from the base layer and the input video stream into the motion estimation unit or both the motion estimation unit and the motion compensation unit in the enhancement layer.