The formulation of the interaction of matter with singular light fields needs special care. In a recent article [Phys.~Rev.~A {bf 91}, 033808 (2015)] we have shown that the Hamiltonian describing the interaction of a twisted light beam having parallel orbital and spin angular momenta with a small object located close to the phase singularity can be expressed only in terms of the electric field of the beam. Here, we complement our studies by providing an interaction Hamiltonian for beams having antiparallel orbital and spin angular momenta. Such beams may exhibit unusually strong magnetic effects. We further extend our formulation to radially and azimuthally polarized beams. The advantages of our formulation are that for all beams the Hamiltonian is written solely in terms of the electric and magnetic fields of the beam and as such it is manifestly gauge-invariant. Furthermore it is intuitive by resembling the well-known expressions in the dipole-electric and dipole-magnetic moment approximations.