We propose a new method for frequency conversion of photons which is both versatile and deterministic. We show that a system with two resonators ultrastrongly coupled to a single qubit can be used to realize both single- and multiphoton frequency-conversion processes. The conversion can be exquisitely controlled by tuning the qubit frequency to bring the desired frequency-conversion transitions on or off resonance. Considering recent experimental advances in ultrastrong coupling for circuit QED and other systems, we believe that our scheme can be implemented using available technology.