Doping of semiconductors by impurity atoms enabled their widespread technological application in micro and opto-electronics. For colloidal semiconductor nanocrystals, an emerging family of materials where size, composition and shape-control offer widely tunable optical and electronic properties, doping has proven elusive. This arises both from the synthetic challenge of how to introduce single impurities and from a lack of fundamental understanding of this heavily doped limit under strong quantum confinement. We develop a method to dope semiconductor nanocrystals with metal impurities providing control of the band gap and Fermi energy. A combination of optical measurements, scanning tunneling spectroscopy and theory revealed the emergence of a confined impurity band and band-tailing. Successful control of doping and its understanding provide n- and p-doped semiconductor nanocrystals which greatly enhance the potential application of such materials in solar cells, thin-film transistors, and optoelectronic devices.