When an atom strongly couples to a cavity, it can undergo coherent vacuum Rabi oscillations. Controlling these oscillatory dynamics quickly relative to the vacuum Rabi frequency enables remarkable capabilities such as Fock state generation and deterministic synthesis of quantum states of light, as demonstrated using microwave frequency devices. At optical frequencies, however, dynamical control of single-atom vacuum Rabi oscillations remains challenging. Here, we demonstrate coherent transfer of optical frequency excitation between a single quantum dot and a cavity by controlling vacuum Rabi oscillations. We utilize a photonic molecule to simultaneously attain strong coupling and a cavity-enhanced AC Stark shift. The Stark shift modulates the detuning between the two systems on picosecond timescales, faster than the vacuum Rabi frequency. We demonstrate the ability to add and remove excitation from the cavity, and perform coherent control of light-matter states. These results enable ultra-fast control of atom-cavity interactions in a nanophotonic device platform.