We present HST WFPC2 V band imaging for 23 Ultraluminous Infrared Galaxies taken from the QDOT redshift survey. The fraction of sources observed to be interacting is 87%. Most of the merging systems show a number of compact `knots, whose colour and brightness differ substantially from their immediate surroundings. Colour maps for nine of the objects show a non-uniform colour structure. Features include blue regions located towards the centres of merging systems which are likely to be areas of enhanced star formation, and compact red regions which are likely to be dust shrouded starbursts or AGN. The host galaxies of the QSOs in the sample were found to be either interacting systems or ellipticals. Our data shows no evidence that ULIRGs are a simple transition stage between galaxy mergers and QSOs. We propose an alternative model for ULIRGs based on the morphologies in our sample and previous N-body simulations. Under this model ULIRGs as a class are much more diverse than a simple transition between galaxy merger and QSO. The evolution of IR power source and merger morphology in ULIRGs is driven solely by the local environment and the morphologies of the merger progenitors.