A voice processing apparatus for performing voiceprint recognition processing with high accuracy even in the case where a plurality of conference participants speak at a time in a conference; wherein a bi-directional telephonic communication portion receives as an input respective voice signals from a plurality of microphones, selects one microphone based on the input voice signals, and outputs a voice signal from the microphone; a voiceprint recognition portion 322 performs voiceprint recognition based on the input voice signal in voiceprint recognizable period, and stores voiceprint data successively in a buffer; and a CPU takes out voiceprint data successively from the buffer, checking against voiceprint data stored in a voiceprint register, specifies a speaker, and processes the voice signal output from the bi-directional telephonic communication portion by associating the same with the speaker.