A variable-size jitter buffer is used to store information associated with a voice signal, facsimile signal or other received signal in a receiver of a packet-based communication system. The receiver determines an appropriate adjustment time for making an adjustment to the size of the buffer based at least in part on a result of a signal detection operation performed on the received signal. For example, in the case of a received voice signal, the determined adjustment time may be a time at which a state machine associated with a speech detector is in a "no speech" state. If the actual buffer size at the determined adjustment time is not within a designated range of a target computed at least in part based on one or more jitter measurements, the buffer size is adjusted at the determined adjustment time, e.g., by an amount representative of the difference between the actual buffer size and the target. The invention provides low-delay and low-complexity jitter buffering particularly well suited for use in an Internet Protocol (IP) receiver of a voice-over-IP system.