Theoretical calculations of core electron binding energies are important for aiding the interpretation of experimental core level photoelectron spectra. In previous work, the $Delta$-Self-Consistent-Field ($Delta$-SCF) method based on density functional theory has been shown to yield highly accurate 1s and 2p binding energies in free molecules. However, most experimental work is concerned with solids, not gases. In this study, we demonstrate the application of the all-electron $Delta$-SCF method to periodic systems. A consideration of the experimentally accessible points of reference leads to the definition of a core electron binding energy in a solid as the difference between the total energies of two $N-1$ electron systems: one with an explicit, localized core hole, and one with an electron removed from the highest occupied state. The calculation of each of these quantities is addressed. In addition, the analogy between a localized core hole and a charged defect in a solid is highlighted, and the extrapolation of calculated core electron binding energies to the infinite supercell limit is discussed. It is found that the extrapolated values of the core electron binding energies from periodic $Delta$-SCF calculations agree well with experimental results for both metallic and insulating systems, with a mean absolute error of 0.24 eV for the 15 core levels considered in this study.