A summary of a captured document image is produced on the basis of detected handwritten annotations made to a document prior to image capture. The scanned (or otherwise captured) image is processed to detect annotations made to the document prior to scanning. The detected annotations can be used to identify features, or text, for use to summarize that document. Additionally, or alternatively, the detected annotations in one document can be used to identify features, or text, for use to summarize a different document. The summary may be displayed in expandable detail levels.