Apparatus for generating sequences of elements including at least one task unit, each of which has an upper and a lower neural network connected in a hierarchical relationship and is operable to output a sequence of elements. Each of the upper and lower neural networks is a class of temporal neural networks having an infinite number of internal states.