In a non-volatile memory, the initiation of program verification is adaptively set so that programming time is decreased. In one approach, non-volatile storage elements are programmed based on a lower page of data to have a voltage threshold (V.sub.TH) that falls within a first V.sub.TH distribution or a higher, intermediate V.sub.TH distribution. Subsequently, the non-volatile storage elements with the first V.sub.TH distribution either remain there, or are programmed to a second V.sub.TH distribution, based on an upper page of data. The non-volatile storage elements with the intermediate V.sub.TH distribution are programmed to third and fourth V.sub.TH distributions. The non-volatile storage elements being programmed to the third V.sub.TH distribution are specially identified and tracked. Verification of the non-volatile storage elements being programmed to the fourth V.sub.TH distribution is initiated after one of the identified non-volatile storage elements transitions to the third V.sub.TH distribution from the intermediate V.sub.TH distribution.