Herein disclosed is a memo image managing apparatus, system and method which are adapted to manage and retrieve a large amount of memo images. The memo image managing apparatus that is connectable to a portable terminal capable of performing a memo image exchange and a memo image display comprises: a managing server adapted to manage memo images and a character information recognizing server adapted to recognize character information of the memo image. The managing server is comprised of a memo image accumulating section adapted to accumulate memo images and a memo image retrieving section adapted to retrieve a predetermined memo image from the accumulated memo images in response to a request. The character information recognizing server is comprised of a character information recognizing section adapted to recognize character information from the memo image.