In this paper, a three-dimensional numerical solver is developed for suspensions of rigid and soft particles and droplets in viscoelastic and elastoviscoplastic (EVP) fluids. The presented algorithm is designed to allow for the first time three-dimensional simulations of inertial and turbulent EVP fluids with a large number particles and droplets. This is achieved by combining fast and highly scalable methods such as an FFT-based pressure solver, with the evolution equation for non-Newtonian (including elastoviscoplastic) stresses. In this flexible computational framework, the fluid can be modelled by either Oldroyd-B, neo-Hookean, FENE-P, and Saramito EVP models, and the additional equations for the non-Newtonian stresses are fully coupled with the flow. The rigid particles are discretized on a moving Lagrangian grid while the flow equations are solved on a fixed Eulerian grid. The solid particles are represented by an Immersed Boundary method (IBM) with a computationally efficient direct forcing method allowing simulations of a large numbers of particles. The immersed boundary force is computed at the particle surface and then included in the momentum equations as a body force. The droplets and soft particles on the other hand are simulated in a fully Eulerian framework, the former with a level-set method to capture the moving interface and the latter with an indicator function. The solver is first validated for various benchmark single-phase and two-phase elastoviscoplastic flow problems through comparison with data from the literature. Finally, we present new results on the dynamics of a buoyancy-driven drop in an elastoviscoplastic fluid.