In the ion acceleration by radiation pressure a transverse inhomogeneity of the electromagnetic pulse results in the displacement of the irradiated target in the off-axis direction limiting achievable ion energy. This effect is described analytically within the framework of the thin foil target model and with the particle-in-cell simulations showing that the maximum energy of accelerated ions decreases while the displacement from the axis of the target initial position increases. The results obtained can be applied for optimization of the ion acceleration by the laser radiation pressure with the mass limited targets.