A control device remotely controls a controlled device. The control device includes a display unit, a communication unit and a control unit. The display unit displays a control panel of the controlled device. The control unit produces operation information. The communication unit sends the operation information to the controlled device. The operation information includes a shifting history of a cursor displayed on the control panel. The shifting history includes identification information of buttons depressed by the cursor in a single operation.