We propose a numerical method to compute the inertial modes of a container with near-spherical geometry based on the fully spectral discretisation of the angular and radial directions using spherical harmonics and Gegenbauer polynomial expansion respectively. This allows to solve simultaneously the Poincare equation and the no penetration condition as an algebraic polynomial eigenvalue problem. The inertial modes of an exact oblate spheroid are recovered to machine precision using an appropriate set of spheroidal coordinates. We show how other boundaries that deviate slightly from a sphere can be accommodated for with the technique of equivalent spherical boundary and we demonstrate the convergence properties of this approach for the triaxial ellipsoid.