A method of rapidly dispatching H.323 packets is disclosed, which is executed after an H.225 protocol connection and before a physical connection of audio/video multimedia data delivery are established between the caller and the callee, and uses Network Address & Port Translation (NAPT) to provide a changed connection port number to H.245 packets between a caller and a callee. A unique connection port number selected by the NAPT is applied to change an original connection port number for packets controlled by the H.245 protocol, thereby providing the changed connection port number as a memory index for add or delete operation and further increasing H.323 packet dispatching rate by directly and quickly comparing memory address through the memory index when searching.