A system provides improved security in a streaming media decoder includes decryption of encoded media information at the media layer, within a decoder component of a playback device. A content source, such as an MPEG-4 media stream from a network, or a file on a local storage device, etc. that contain packetized content, and supplies encrypted and encoded media information. For example, digital sound and image information can be conveyed for presentation. A reassembler component is used to initiate decryption of a nominal amount of information needed to perform further processing at the QuickTime player. Encrypted access units are sent from the reassembler to the decoder, where the decoder causes decryption of the access unit information. This approach prevents decrypted, encoded information from being transferred from the reassembler to the decoder. Such decrypted and encoded information is especially susceptible to copying. Instead, the information transferred from the reassembler to the decoder is largely still encrypted. In a preferred embodiment, an Internet Protocol rights management system (IPRM) is used to perform the decryption processing. The reassembler makes requests of the IPRM process to open a decryption session and decrypt selected packets to identify the window rendering size. The decoder makes requests of the IPRM process to decrypt access units within the same decryption session and to close the decryption session. Because content is encrypted on packet boundaries, and when called upon by the decoder, the IPRM component must recreate the packet boundaries from the access unit before decryption is possible. The IPRM system component is able to do so because the encrypted packets contain an IPRM header that is used by the IPRM system to calculate the packet length. After decrypting all the packets, the IPRM system component reconstructs the decrypted access unit and returns decrypted content to the decoder.