Optical unitary converter (OUC) that can convert a set of N mutually orthogonal optical modes into another set of arbitrary N orthogonal modes is expected to be the key device in diverse applications, including the optical communication, deep learning, and quantum computing. While various types of OUC have been demonstrated on photonic integration platforms, its sensitivity against a slight deviation in the waveguide dimension has been the crucial issue in scaling N. Here, we demonstrate that an OUC based on the concept of multi-plane light conversion (MPLC) shows outstanding robustness against waveguide deviations. Moreover, it becomes more and more insensitive to fabrication errors as we increase N, which is in clear contrast to the conventional OUC architecture, composed of 2 $times$ 2 Mach-Zehnder interferometers. The physical origin behind this unique robustness and scalability is studied by considering a generalized OUC configuration. As a result, we reveal that the number of coupled modes in each stage plays an essential role in determining the sensitivity of the entire OUC. The maximal robustness is attained when all-to-all-coupled interferometers are employed, which are naturally implemented in MPLC-OUC.