The quantum spin systems Cu$_2$MBO$_5$ (M = Al, Ga) with the ludwigite crystal structure consist of a structurally ordered Cu$^{2+}$ sublattice in the form of three-leg ladders, interpenetrated by a structurally disordered sublattice with a statistically random site occupation by magnetic Cu$^{2+}$ and nonmagnetic Ga$^{3+}$ or Al$^{3+}$ ions. A microscopic analysis based on density-functional-theory calculations for Cu$_2$GaBO$_5$ reveals a frustrated quasi-two-dimensional spin model featuring five inequivalent antiferromagnetic exchanges. A broad low-temperature $^{11}$B nuclear magnetic resonance points to a considerable spin disorder in the system. In zero magnetic field, antiferromagnetic order sets in below $T_text{N}$ $approx$ 4.1 K and ~2.4 K for the Ga and Al compounds, respectively. From neutron diffraction, we find that the magnetic propagation vector in Cu$_2$GaBO$_5$ is commensurate and lies on the Brillouin-zone boundary in the (H0L) plane, $mathbf{q}_text{m}$ = (0.45 0 -0.7), corresponding to a complex noncollinear long-range ordered structure with a large magnetic unit cell. Muon spin relaxation is monotonic, consisting of a fast static component typical for complex noncollinear spin systems and a slow dynamic component originating from the relaxation on low-energy spin fluctuations. Gapless spin dynamics in the form of a diffuse quasielastic peak is also evidenced by inelastic neutron scattering. Most remarkably, application of a magnetic field above 1 T destroys the static long-range order, which is manifested in the gradual broadening of the magnetic Bragg peaks. We argue that such a crossover from a magnetically long-range ordered state to a spin-glass regime may result from orphan spins on the structurally disordered magnetic sublattice, which are polarized in magnetic field and thus act as a tuning knob for field-controlled magnetic disorder.