Acoustophoresis deals with the manipulation of sub-wavelength scatterers in an incident acoustic field. The geometric details of manipulated particles are often neglected by replacing them with equivalent symmetric geometries such as spheres, spheroids, cylinders or disks. It has been demonstrated that geometric asymmetry, represented by Willis coupling terms, can strongly affect the scattering of a small object, hence neglecting these terms may miss important force contributions. In this work, we present a generalized formalism of acoustic radiation force and radiation torque based on the polarizability tensor, where Willis coupling terms are included to account for geometric asymmetry. Following Gorkovs approach, the effects of geometric asymmetry are explicitly formulated as additional terms in the radiation force and torque expressions. By breaking the symmetry of a sphere along one axis using intrusion and protrusion, we characterize the changes in the force and torque in terms of partial components, associated with the direct and Willis Coupling coefficients of the polarizability tensor. We investigate in detail the cases of standing and travelling plane waves, showing how the equilibrium positions and angles are shifted by these additional terms. We show that while the contributions of asymmetry to the force are often negligible for small particles, these terms greatly affect the radiation torque. Our presented theory, providing a way of calculating radiation force and torque directly from polarizability coefficients, shows that in general it is essential to account for shape of objects undergoing acoustophoretic manipulation, and this may have important implications for applications such as the manipulation of biological cells.