Quite recently I have proposed a nonperturbative dynamical effective field model (DEFM) to quantitatively describe the dynamics of interacting ferrofluids. Its predictions compare very well with the results from simulations. In this paper I put the DEFM on firm theoretical ground by deriving it within the framework of dynamical density functional theory (DDFT), in which the relevant part of correlation-induced free energy is approximated by a function of the instantaneous magnetization. The DEFM is generalized to inhomogeneous finite-size samples for which the macroscopic and mesoscopic scale separation is nontrivial due to the presence of long-range dipole-dipole interactions. The demagnetizing field naturally emerges from microscopic considerations and is consistently accounted for. The resulting particle dynamics on the mesoscopic scale only involves macroscopically local quantities such as local magnetization and Maxwell field. Nevertheless, the local demagnetizing field essentially couples to magnetization at distant macroscopic locations. Thus, a two-scale parallel algorithm, involving information transfer between different macroscopic locations, can be applied to fully resolve particle rotational dynamics in an inhomogeneous sample. I also derive the DEFM for polydisperse ferrofluids, in which the dynamics of particles belonging to different species can be strongly coupled to each other. I discuss the underlying assumptions in obtaining a thermodynamically consistent polydisperse magnetization relaxation equation, which is of the same generic form as that for monodisperse ferrofluids. The theoretical advances presented in this paper are important for both qualitative understanding and quantitative modeling of ferrofluid dynamics.