We have carried out full imaging simulation studies to explore the impact of frequency standards in millimeter and sub-millimeter Very Long Baseline Interferometry (VLBI), focusing on the coherence time and sensitivity. In particular, we compare the performance of the H-maser, traditionally used in VLBI, to that of ultra-stable cryocooled sapphire oscillators over a range of observing frequencies, weather conditions and analysis strategies. Our simulations show that at the highest frequencies, the losses induced by H-maser instabilities are comparable to those from high quality tropospheric conditions. We find significant benefits in replacing H-masers with cryocooled sapphire oscillator based frequency references in VLBI observations at frequencies above 175 GHz in sites which have the best weather conditions; at 350 GHz we estimate a 20-40% increase in sensitivity, over that obtained when the sites have H-masers, for coherence losses of 20-10%, respectively. Maximum benefits are to be expected by using colocated Water Vapour Radiometers for atmospheric correction. In this case, we estimate a 60-120% increase in sensitivity over the H-maser at 350 GHz.