For changing a segmentation of an audio piece after a segment class assignment, at first a short segment is selected, which has a length shorter than a predetermined minimum length. This short segment is preferably merged with the corresponding successor segment or predecessor segment using information on a segment class membership of the short segment itself, but also the successor segment or the predecessor segment, in order to obtain a changed segmentation of the audio signal. With this, a not over-segmented segment representation of the audio signal is obtained, which further includes all audio information, i.e. is not a representation of the audio piece with holes.