Systems and methods for creating aesthetically pleasing visual image summaries from a collection of input images. The systems and methods determine a visual image summary by creating candidate partial layouts, assigning penalty scores to the candidate partial layouts based on a number of evaluation criteria, evaluating the penalty scores for the candidate partial layouts, and building upon the candidate partial layouts, beginning with the lowest scoring candidate partial layout first, to form complete layouts. The resultant visual image summary fits within a user defined area, maintains the order of the input images, and contains a relative small amount of aesthetically undesirable qualities.