For a symmetry consistent theoretical description of the multiferroic phase of Ba$_2$CoGe$_2$O$_7$ a precise knowledge of its crystal structure is a prerequisite. In our previous synchrotron X-ray diffraction experiment on multiferroic Ba$_2$CoGe$_2$O$_7$ at room temperature we found forbidden reflections that favour the tetragonal-to-orthorhombic symmetry lowering of the titled compound. Here, we report the results of room-temperature single-crystal diffraction studies with both hot and cold neutrons to differentiate between the real symmetry lowering and multiple diffraction (the Renninger effect). A comparison of the experimental multiple diffraction patterns with simulated ones rules out the symmetry lowering. Thus, the structural model based on the tetragonal space group $Pbar{4}2_1m$ was selected to describe the Ba$_2$CoGe$_2$O$_7$ symmetry at room temperature. The precise structural parameters from neutron diffraction at 300K are presented and compared with the previous X-ray diffraction results.