A semiconductor memory device includes a pair of memory sub arrays and a control signal generating circuit. The pair of memory sub arrays shares a sense amplifier, and each of the pair of memory sub arrays has a plurality of memory cells arranged in a matrix. Each of columns of the matrix is connected to a pair of bit lines, and each of rows of the matrix is connected to a word line. The control signal generating circuit sequentially outputs first and second refresh start signals within an operation time to an external refresh command in response to an internal refresh command. A first refreshing operation is carried out to first memory cells connected to a first word line of one of the memory sub arrays in response to the first refresh start signal, and a second refreshing operation is carried out to second memory cells connected to a second word line different from the first word line in the memory sub array in response to the second refresh start signal.