It has been theoretically predicted that light carrying orbital angular momentum, or twisted light, can be tuned to have a strong magnetic-field component at optical frequencies. We here consider the interaction of these peculiar fields with a semiconductor quantum dot and show that the magnetic interaction results in new types of optical transitions. In particular, a single pulse of such twisted light can drive light-hole-to-conduction band transitions that are cumbersome to produce using conventional Gaussian beams or even twisted light with dominant electric fields.