We present a novel computational modeling framework to numerically investigate fluid-structure interaction in viscous fluids using the phase field embedding method. Each rigid body or elastic structure immersed in the incompressible viscous fluid matrix, grossly referred to as the particle in this paper, is identified by a volume preserving phase field. The motion of the particle is driven by the fluid velocity in the matrix for passive particles or combined with its self-propelling velocity for active particles. The excluded volume effect between a pair of particles or between a particle and the boundary is modeled by a repulsive potential force. The drag exerted to the fluid by a particle is assumed proportional to its velocity. When the particle is rigid, its state is described by a zero velocity gradient tensor within the nonzero phase field that defines its profile and a constraining stress exists therein. While the particle is elastic, a linear constitutive equation for the elastic stress is provided within the particle domain. A hybrid, thermodynamically consistent hydrodynamic model valid in the entire computational domain is then derived for the fluid-particle ensemble using the generalized Onsager principle accounting for both rigid and elastic particles. Structure-preserving numerical algorithms are subsequently developed for the thermodynamically consistent model. Numerical tests in 2D and 3D space are carried out to verify the rate of convergence and numerical examples are given to demonstrate the usefulness of the computational framework for simulating fluid-structure interactions for passive as well as self-propelling active particles in a viscous fluid matrix.