Sound signal indicative of a human voice or musical tone is input, and the pitch of the input sound signal is detected. Then, a scale note pitch is determined which is nearest to the detected pitch of the input sound signal. In the meantime, a scale note pitch of an additional sound or harmony sound to be added to the input sound is specified in accordance with a harmony mode selected by a user. The scale note pitch of the additional sound to be generated is modified in accordance with a difference between the determined scale note pitch and the detected pitch of the input sound signal. Because the additional sound is generated with the modified pitch, it can appropriately follow a variation in the pitch of the input sound to be in harmony with the input sound, rather than exactly agreeing with the scale note pitch. As another example, reference scale note pitch data may be supplied, instead of the scale note pitch nearest to the detected pitch of the input sound signal being determined in the above-mentioned manner.