The present invention provides multi-user interaction for multimedia communication. In one embodiment, a process for multi-user interaction for multimedia communication includes generating a message on a local user machine, the message including object-based media data (i.e., streamed, digital audio data or streamed, digital video data or both), and transmitting the message to a remote user machine, in which the local user machine displays a scene that includes the object-based media data, the scene being shared by the local user machine and the remote user machine. The remote user machine constructs the message using a message handler class. In one embodiment, the multi-user inter-action for multimedia communication is an extension to MPEG-4 Version-1.