We report on the first calculation of the scattering length A_{K^-d} based on a relativistic three-body approach where the two-body input amplitudes coupled to the Kbar N channels have been obtained with the chiral SU(3) constraint, but with isospin symmetry breaking effects taken into account. Results are compared with a recent calculation applying a similar set of two-body amplitudes,based on the fixed center approximation, considered as a good approximation for a loosely bound target, and for which we find significant deviations from the exact three-body results. Effects of the hyperon-nucleon interaction, and deuteron $D$-wave component are also evaluated.