The crystal structure of the candidate topological superconductor Cu$_x$Bi$_2$Se$_3$ was studied by single-crystal neutron diffraction using samples obtained by inserting the Cu dopant electrochemically. Neither structural refinements nor calculated scattering-density maps find a significant occupation of Cu at the intercalation site between the quintuple layers of Bi$_2$Se$_3$. Following Bragg reflection intensities as function of temperature, there is no signature of a structural phase transition between 295 and 2 K. However, the analysis of large sets of Bragg reflections indicates a small structural distortion breaking the rotational axis due to small displacements of the Bi ions.