We investigate the evolution and origin of small-scale chromospheric swirls by analyzing numerical simulations of the quiet solar atmosphere, using the radiative magnetohydrodynamic code CO$5$BOLD. We are interested in finding their relation with magnetic field perturbations and in the processes driving their evolution. For the analysis, the swirling strength criterion and its evolution equation are applied in order to identify vortical motions and to study their dynamics. We introduce a new criterion, the magnetic swirling strength, which allows us to recognize torsional perturbations in the magnetic field. We find a strong correlation between swirling strength and magnetic swirling strength, in particular in intense magnetic flux concentrations, which suggests a tight relation between vortical motions and torsional magnetic field perturbations. Furthermore, we find that swirls propagate upward with the local Alfven speed as unidirectional swirls, in the form of pulses, driven by magnetic tension forces alone. In the photosphere and low chromosphere, the rotation of the plasma co-occurs with a twist in the upwardly directed magnetic field that is in the opposite direction of the plasma flow. All together, these are characteristics of torsional Alfven waves. We also find indications of an imbalance between the hydrodynamic and magnetohydrodynamic baroclinic effects being at the origin of the swirls. At the base of the chromosphere, we find a net upwardly directed Poynting flux, which is mostly associated with large and complex swirling structures that we interpret as the superposition of various small-scale vortices. We conclude that the ubiquitous swirling events observed in simulations are tightly correlated with perturbations of the magnetic field. At photospheric and chromospheric levels, they form Alfven pulses that propagate upward and may contribute to chromospheric heating.