A method and system for increasing the spectral efficiency of binary coded signals includes encoding an input binary bit sequence such that the input binary bit sequence is converted into a series of rectangular pulses having varying repetition rates. A continuous wave carrier signal is then modulated via a control signal representative of the encoded signal. The modulated optical signal is filtered by a narrow band optical filter to generate a minimum shift keying (MSK) encoded optical signal. In accordance with the present invention, an input binary bit sequence is encoded via a minimum shift keying (MSK) modulation format to improve the spectral efficiency of a binary coded digital signal and to contemporaneously limit distortion of the optical signal induced by optical nonlinear effects during transmission in an optical transmission system.