We apply the point form of relativistic quantum mechanics to develop a Poincare invariant coupled-channel formalism for two-particle systems interacting via one-particle exchange. This approach takes the exchange particle explicitly into account and leads to a generalized eigenvalue equation for the Bakamjian-Thomas type mass operator of the system. The coupling of the exchange particle is derived from quantum field theory. As an illustrative example we consider vector mesons within the chiral constituent quark model in which the hyperfine interaction between the confined quark-antiquark pair is generated by Goldstone-boson exchange. We study the effect of retardation in the Goldstone-boson exchange by comparing with the commonly used instantaneous approximation. As a nice physical feature we find that the problem of a too large $rho$-$omega$ splitting can nearly be avoided by taking the dynamics of the exchange meson explicitly into account.