We have observed nonlinear transduction of the thermomechanical motion of a nanomechanical resonator when detected as laser transmission through a sideband unresolved optomechanical cavity. Nonlinear detection mechanisms are of considerable interest as special cases allow for quantum nondemolition measurements of the mechanical resonators energy. We investigate the origin of the nonlinearity in the optomechanical detection apparatus and derive a theoretical framework for the nonlinear signal transduction, and the optical spring effect, from both nonlinearities in the optical transfer function and second order optomechanical coupling. By measuring the dependence of the linear and nonlinear signal transduction -- as well as the mechanical frequency shift -- on laser detuning from optical resonance, we provide estimates of the contributions from the linear and quadratic optomechanical couplings.