In one embodiment, the present invention includes a cache memory, which may be a sequential cache, having multiple banks. Each of the banks includes a data array, a decoder coupled to the data array to select a set of the data array, and a sense amplifier. Only a bank to be accessed may be powered, and in some embodiments early way information may be used to maintain remaining banks in a power reduced state. In some embodiments, clock gating may be used to maintain various components of the cache memory in a power reduced state. Other embodiments are described and claimed.