An encoder/decoder uses "self-referencing" frames. For example, a second B-field in a current frame references the first B-field from the current frame in motion compensated prediction. Allowing the first B-field in a frame to act as a reference for the second B-field in the frame allows more accurate prediction of the second B-field, while also preserving the temporal scalability benefits of having B-fields in the current frame.