A system for learning an attention model for an image based on user navigation actions while viewing the image is provided. An attention model learning system generates an initial attention model based on static features derived from the image. The learning system logs the actions of users as they view the image. The learning system identifies from the actions of the users those areas of the image that may be of user interest: After the learning system identifies areas that may be of user interest, the learning system attempts to identify attention objects based on those identified areas. After identifying the user attention objects, the learning system updates the attention model for the image to reflect the additional information provided by the user attention objects.