We explore the field-temperature phase diagram of the XY pyrochlore antiferromagnet Er$_2$Ti$_2$O$_7$, by means of magnetization and neutron diffraction experiments. Depending on the field strength and direction relative to the high symmetry cubic directions $[001], [1bar{1}0]$ and $[111]$, the refined field induced magnetic structures are derived from the zero field $psi_2$ and $psi_3$ states of the $Gamma_5$ irreducible representation which describes the ground state of XY pyrochlore antiferromagnets. At low field, domain selection effects are systematically at play. In addition, for $[001]$, a phase transition is reported towards a $psi_3$ structure at a characteristic field $H_c^{001}=$ 43 mT. For $[1bar{1}0]$ and $[111]$, the spins are continuously tilted by the field from the $psi_2$ state, and no phase transition is found while domain selection gives rise to sharp anomalies in the field dependence of the Bragg peaks intensity. For $[1bar{1}0]$, these results are confirmed by high resolution inelastic neutron scattering experiments, which in addition allow us to determine the field dependence of the spin gap. This study agrees qualitatively with the scenario proposed theoretically by Maryasin {it et al.} [Phys. Rev. B {bf 93}, 100406(R) (2016)], yet the strength of the field induced anisotropies is significantly different from theory.