Inspired by recent discussions of inverse magnetic catalysis in the literature, we examine the effects of a uniform external magnetic field on the chiral phase transition in quenched ladder QED at nonzero chemical potential. In particular, we study the behaviour of the effective potential as the strength of the magnetic field is varied while the chemical potential is held constant. For a certain range of the magnetic field, the effective potential develops a local maximum. Inverse magnetic catalysis is observed at this maximum, whereas the usual magnetic catalysis is observed at the true minimum of the effective potential.