A system and method are provided for improving the accuracy of speaker authentication by combining the results of multiple verification sources using statistical modeling. A statistical classifier is built on top of the underlying verification sources. The classifier combines the multiple verification inputs with associated models of the respective verification technologies underlying the verification inputs. The classifier functions to classify a claimant as a true speaker or impostor based upon the verification inputs, the model or models and, potentially, additional information related to the context in which the classifier is being used.