A system and method are disclosed for streaming scalable video data over a variable-bandwidth network such as a packet-based one. In other words, the number of bits (for FGS) or sub-layers (for discrete multi-layer scalability) is chosen to be transmitted for each enhancement layer frame. Further, these bits or sub-layers are also partitioned over multiple packets if being sent over a packet-based network. Further, a windowing function is used to smooth the variation in the number of bits or sub-layers transmitted for each frame in situations where the network bandwidth is varying.