An approach to multi-lingual speech recognition that permits different words in an utterance to be from different languages. Words from different languages are represented using different sets of sub-word units that are each associate with the corresponding language. Despite the use of different sets of sub-word units, the approach enables use of cross-word context at boundaries between words from different languages (cross-language context) to select appropriate variants of the sub-word units to match the context.