A method for lexically analyzing an input stream including invoking a multipurpose lexical analyzer, wherein invoking the multipurpose lexical analyzer comprises examining a parameter setting, scanning the input stream to obtain a token using a token definition, and determining whether to ignore the token using the parameter setting, returning the token to a calling process if the token is to be processed.