We derive renormalization group equations which allow us to treat order parameter fluctuations near quantum phase transitions in cases where an expansion in powers of the order parameter is not possible. As a prototypical application, we analyze the nematic transition driven by a d-wave Pomeranchuk instability in a two-dimensional electron system. We find that order parameter fluctuations suppress the first order character of the nematic transition obtained at low temperatures in mean-field theory, so that a continuous transition leading to quantum criticality can emerge.