Wireless backhaul is considered to be the key part of the future wireless network with dense small cell traffic and high capacity demand. In this paper, we focus on the design of a high spectral efficiency line-of-sight (LoS) multiple-input multiple-output (MIMO) system for millimeter wave (mmWave) backhaul using dual-polarized frequency division duplex (FDD). High spectral efficiency is very challenging to achieve for the system due to various physical impairments such as phase noise (PHN), timing offset (TO) as well as the poor condition number of the LoS MIMO. In this paper, we propose a holistic solution containing TO compensation, PHN estimation, precoder/decorrelator optimization of the LoS MIMO for wireless backhaul, and the interleaving of each part. We show that the proposed solution has robust performance with end-to-end spectral efficiency of 60 bits/s/Hz for 8x8 MIMO.