The present invention relates to an audio data synthesis system for sequentially processing a first predetermined number of audio data to synthesize a digital audio signal cumulatively. The system comprises a first memory, a first processor, an audio data processing unit, and a second memory. The first memory is for storing a plurality of audio data. The first processor is for generating an audio processing request for requesting to process a second predetermined number of audio data. The audio data processing unit is for receiving the audio processing request, accessing the second predetermined number of audio data stored in the first memory, and calculating every two neighboring audio data to get data processing values, and after calculating all the second predetermined number of audio data, then obtaining a third predetermined number of data processing values. The second memory is for storing the third predetermined number of audio data.