In this paper, probabilistic shaping is numerically and experimentally investigated for increasing the transmission reach of wavelength division multiplexed (WDM) optical communication system employing quadrature amplitude modulation (QAM). An optimized probability mass function (PMF) of the QAM symbols is first found from a modified Blahut-Arimoto algorithm for the optical channel. A turbo coded bit interleaved coded modulation system is then applied, which relies on many-to-one labeling to achieve the desired PMF, thereby achieving shaping gain. Pilot symbols at rate at most 2% are used for synchronization and equalization, making it possible to receive input constellations as large as 1024QAM. The system is evaluated experimentally on a 10 GBaud, 5 channels WDM setup. The maximum system reach is increased w.r.t. standard 1024QAM by 20% at input data rate of 4.65 bits/symbol and up to 75% at 5.46 bits/symbol. It is shown that rate adaptation does not require changing of the modulation format. The performance of the proposed 1024QAM shaped system is validated on all 5 channels of the WDM signal for selected distances and rates. Finally, it was shown via EXIT charts and BER analysis that iterative demapping, while generally beneficial to the system, is not a requirement for achieving the shaping gain.