The invention performs handwriting recognition using mixtures of Bayesian networks. A mixture of Bayesian networks (MBN) consists of plural hypothesis-specific Bayesian networks (HSBNs) having possibly hidden and observed variables. A common external hidden variable is associated with the MBN, but is not included in any of the HSBNs. Each HSBN models the world under the hypothesis that the common external hidden variable is in a corresponding one of its states. The MBNs encode the probabilities of observing the sets of visual observations corresponding to a handwritten character. Each of the HSBNs encodes the probabilities of observing the sets of visual observations corresponding to a handwritten character and given a hidden common variable being in a particular state.