Integrated quantum photonic applications, providing physially guaranteed communications security, sub-shot-noise measurement, and tremendous computational power, are nearly within technological reach. Silicon as a technology platform has proven formibable in establishing the micro-electornics revoltution, and it might do so again in the quantum technology revolution. Silicon has has taken photonics by storm, with its promise of scalable manufacture, integration, and compatibility with CMOS microelectronics. These same properties, and a few others, motivate its use for large-scale quantum optics as well. In this article we provide context to the development of quantum optics in silicon. We review the development of the various components which constitute integrated quantum photonic systems, and we identify the challenges which must be faced and their potential solutions for silicon quantum photonics to make quantum technology a reality.