A system and method interpolates a full color image from an array of single color sensors. Each sensor measures a single color, e.g., red, green or blue. The measured color values are stored as an array of data. For each data element, a plurality, e.g., four, gradients are computed that specify the color and/or luminance difference along different linear paths extending through the data element. At least some of the gradients are computed using a first order color differential and a second order color or luminance differential along the selected path. One or more gradients are then selected based on a comparison of the gradients to a computed threshold. Using the data elements along the path of the selected gradient or gradients, the missing color values are interpolated. The interpolation algorithm also utilizes first and second order color differentials.