A method for segmenting an image using a background-based segmentation process is provided. A document image (102) is low-pass filtered and decimated. The decimated image is processed at low resolution by a low-resolution segmentation (104) stage. Segmentation results include identification of a main background and one or more objects. Objects that cannot be classified in text or picture classes are further segmented into a local background and smaller objects. This process is reiterated until all objects are classified in text or picture classes. The results are overlaid on the image (102) during an original-resolution refinement (106) stage to refine the segmentation.