A method of accessing internal memory of a processor and the device thereof. The method employs a bank swapping mechanism for the processing unit of a processor and a direct memory access controller to simultaneously access different memory units in internal memory. The processing unit can continuously access and process data in the internal memory to optimize its efficiency. In the device, the processing unit of a processor and a direct memory access controller are coupled to internal memory through a switching circuit, the switch of which enables the processing unit and the direct memory access controller to access different memory units in the internal memory. Therefore, the processing unit can continuously access and process data in the internal memory to optimize its efficiency.