We present a study of the Rayleigh-Taylor unstable regime of accretion onto rotating magnetized stars in a set of high grid resolution three-dimensional (3D) magnetohydrodynamic (MHD) simulations performed in low-viscosity discs. We find that the boundary between the stable and unstable regimes is determined almost entirely by the fastness parameter omega_s=Omega_star/Omega_K(r_m), where Omega_star is the angular velocity of the star and Omega_K(r_m) is the angular velocity of the Keplerian disc at the disc-magnetosphere boundary r=r_m. We found that accretion is unstable if omega_s < 0.6. Accretion through instabilities is present in stars with different magnetospheric sizes. However, only in stars with relatively small magnetospheres, r_m/R_star < 7, do the unstable tongues produce chaotic hot spots on the stellar surface and irregular light-curves. At even smaller values of the fastness parameter, omega_s < 0.45, multiple irregular tongues merge, forming one or two ordered unstable tongues that rotate with the angular frequency of the inner disc. This transition occurs in stars with even smaller magnetospheres, r_m/R_star < 4.2. Most of our simulations were performed at a small tilt of the dipole magnetosphere, Theta=5 degrees, and a small viscosity parameter alpha=0.02. Test simulations at higher alpha values show that many more cases become unstable, and the light-curves become even more irregular. Test simulations at larger tilts of the dipole Theta show that instability is present, however, accretion in two funnel streams dominates if Theta > 15 degrees. The results of these simulations can be applied to accreting magnetized stars with relatively small magnetospheres: Classical T Tauri stars, accreting millisecond X-ray pulsars, and cataclysmics variables.