Geometrical frustration among interacting spins combined with strong quantum fluctuations destabilize long-range magnetic order in favor of more exotic states such as spin liquids. By following this guiding principle, a number of spin liquid candidate systems were identified in quasi-two-dimensional (quasi-2D) systems. For 3D, however, the situation is less favorable as quantum fluctuations are reduced and competing states become more relevant. Here we report a comprehensive study of thermodynamic, magnetic and dielectric properties on single crystalline and pressed-powder samples of PbCuTe$_2$O$_6$, a candidate material for a 3D frustrated quantum spin liquid featuring a hyperkagome lattice. Whereas the low-temperature properties of the powder samples are consistent with the recently proposed quantum spin liquid state, an even more exotic behaviour is revealed for the single crystals. These crystals show ferroelectric order at $T_{text{FE}} approx 1,text{K}$, accompanied by strong lattice distortions, and a modified magnetic response -- still consistent with a quantum spin liquid -- but with clear indications for quantum critical behaviour.