We have proposed and demonstrated a general and scalable scheme for programmable unitary gates. Our method is based on matrix decomposition into diagonal and Fourier factors. Thus, we are able to construct arbitrary matrix operators only by diagonal matrices alternately acting on two photonic encoding bases that belong to a Fourier transform pair. Thus, the technical difficulties to implement arbitrary unitary operators are significantly reduced. As examples, two protocols for optical OAM and path domain are considered to verify our proposal and evaluate the performance. For OAM domain, unitary matrices with dimensionality up to 6*6 are achieved and discussed with the numerical simulations. For path domain, an average fidelity of 0.98 is evaluated through 80 experimental results with dimensionality of 3*3. Furthermore, our proposal is also potential to provide a quantum operation protocol for any other photonic domain, if there exists the corresponding transformed domain.