A number of recent studies indicate that the charge conduction of the LaAlO$_3$/SrTiO$_3$ interface at low temperature is confined to filaments which are linked to structural domain walls in the SrTiO$_3$ with drastic consequences for example for the temperature dependence of local transport properties. We demonstrate that as a consequences of this current carrying filaments on the nano-scale the magnetotransport properties of the interface are highly anisotropic. Our magnetoresistance measurements reveal that the magnetoresistance in different nanostructures ($<500nm$) is random in magnitude and sign, respectively. Warming up nanostructures above the structural phase transition temperature (105K) results in the significant change in MR. Even a sign change of the magnetoresistance is possible. The results suggest that domain walls that are differently oriented with respect to the surface exhibit different respective magnetoresistance and the total magnetoresistance is a result of a random domain wall pattern formed during the structural phase transition in the SrTiO$_3$ at cool down.