The present invention relates to a system and methodology providing layered probabilistic representations for sensing, learning, and inference from multiple sensory streams at multiple levels of temporal granularity and abstraction. The methods facilitate robustness to subtle changes in environment and enable model adaptation with minimal retraining. An architecture of Layered Hidden Markov Models (LHMMs) can be employed having parameters learned from stream data and at different periods of time, wherein inferences can be determined relating to context and activity from perceptual signals.