A system and method for modifying images has an ability to assign color intensity values to pixels exposed during image manipulation operations. The system and method for modifying images comprises means for using the color intensity values of remaining pixels in the original image, and for assigning color values to the exposed pixels that are similar to those of the surrounding pixels so that the exposed regions blends smoothly with the surrounding region. The means for assigning color values assigns the value of color intensity based on the color intensity value function determined at a location of a pixel. The color intensity value function used to assign values to exposed pixels is determined by fitting a function to the known color intensity values of pixels in the boundary regions of the exposed surrounding pixels. The color intensity value function is approximated using energy minimization along with boundary conditions.