We present a very efficient solver for the general Anderson impurity problem. It is based on the perturbation around a solution obtained from exact diagonalization using a small number of bath sites. We formulate a perturbation theory which is valid for both weak and strong coupling and interpolates between these limits. Good agreement with numerically exact quantum Monte-Carlo results is found for a single bath site over a wide range of parameters. In particular, the Kondo resonance in the intermediate coupling regime is well reproduced for a single bath site and the lowest order correction. The method is particularly suited for low temperatures and alleviates analytical continuation of imaginary time data due to the absence of statistical noise compared to quantum Monte-Carlo impurity solvers.