We investigate the break-up of Newtonian/viscoelastic droplets in a viscoelastic/Newtonian matrix under the hydrodynamic conditions of a confined shear flow. Our numerical approach is based on a combination of Lattice-Boltzmann models (LBM) and Finite Difference (FD) schemes. LBM are used to model two immiscible fluids with variable viscosity ratio (i.e. the ratio of the droplet to matrix viscosity); FD schemes are used to model viscoelasticity, and the kinetics of the polymers is introduced using constitutive equations for viscoelastic fluids with finitely extensible non-linear elastic dumbbells with Peterlins closure (FENE-P). We study both strongly and weakly confined cases to highlight the role of matrix and droplet viscoelasticity in changing the droplet dynamics after the startup of a shear flow. Simulations provide easy access to quantities such as droplet deformation and orientation and will be used to quantitatively predict the critical Capillary number at which the droplet breaks, the latter being strongly correlated to the formation of multiple neckings at break-up. This study complements our previous investigation on the role of droplet viscoelasticity (A. Gupta & M. Sbragaglia, {it Phys. Rev. E} {bf 90}, 023305 (2014)), and is here further extended to the case of matrix viscoelasticity.