The position measurement system includes: an image capturing unit capturing first reference points which are three reference points on a plane arranged on a target and whose positional relation with each other is specified, and a second reference point which is a single reference point at a distance away from the plane and whose positional relation with the first reference points is specified; an identifying unit identifying images of the first and second reference points based on the positional relation among the images; and a calculating unit calculating a three-dimensional position and triaxial angles of the target based on the positional relation between the images of the first and second reference points. The calculating unit specifies the plane including the first reference points on the target from the images of the first reference points and a normal direction of the plane according to the image of the second reference point.