A packer for downhole use features interacting elements of swelling material. Preferably the elements are in contact for relative movement from an initial diameter for run in. As the elements swell, they move with respect to each other to enlarge the diameter of the assembly so that a sealing contact is made. Each element exerts a residual force on the adjacent element to enhance the seal. Each element is preferably coated with a material that allows well fluids to reach the swelling material and then later to stiffen and become impervious from exposure to such fluids. The assembly can be covered for run in to delay the onset of expansion until the target depth is reached for the packer to be set.