We demonstrate a method for general linear optical networks that allows one to factorize any SU($n$) matrix in terms of two SU($n-1)$ blocks coupled by an SU(2) entangling beam splitter. The process can be recursively continued in an efficient way, ending in a tidy arrangement of SU(2) transformations. The method hinges only on a linear relationship between input and output states, and can thus be applied to a variety of scenarios, such as microwaves, acoustics, and quantum fields.