Using the momentum average approximation we study the importance of adding higher-than-linear terms in the electron-phonon coupling on the properties of single polarons described by a generalized Holstein model. For medium and strong linear coupling, even small quadratic electron-phonon coupling terms are found to lead to very significant quantitative changes in the properties of the polaron, which cannot be captured by a linear Holstein Hamiltonian with renormalized parameters. We argue that the bi-polaron phase diagram is equally sensitive to addition of quadratic coupling terms if the linear coupling is large. These results suggest that the linear approximation is likely to be inappropriate to model systems with strong electron-phonon coupling, at least for low carrier concentrations.