Smoothed particle hydrodynamics (SPH) is positioned as having ideal conservation properties. When properly implemented, conservation of total mass, energy, and both linear and angular momentum is guaranteed exactly, up to machine precision. This is particularly important for some applications in computational astrophysics, such as binary dynamics, mergers, and accretion of compact objects (neutron stars, black holes, and white dwarfs). However, in astrophysical applications that require the inclusion of gravity, calculating pairwise particle interactions becomes prohibitively expensive. In the Fast Multipole Method (FMM), they are, therefore, replaced with symmetric interactions between distant clusters of particles (contained in the tree nodes) Although such an algorithm is linear momentum-conserving, it introduces spurious torques that violate conservation of angular momentum. We present a modification of FMM that is free of spurious torques and conserves angular momentum explicitly. The new method has practically no computational overhead compared to the standard FMM.