Deep underground environments are ideal for low background searches due to the attenuation of cosmic rays by passage through the earth. However, they are affected by backgrounds from $gamma$-rays emitted by $^{40}$K and the $^{238}$U and $^{232}$Th decay chains in the surrounding rock. The LUX-ZEPLIN (LZ) experiment will search for dark matter particle interactions with a liquid xenon TPC located within the Davis campus at the Sanford Underground Research Facility, Lead, South Dakota, at the 4,850-foot level. In order to characterise the cavern background, in-situ $gamma$-ray measurements were taken with a sodium iodide detector in various locations and with lead shielding. The integral count rates (0--3300~keV) varied from 596~Hz to 1355~Hz for unshielded measurements, corresponding to a total flux in the cavern of $1.9pm0.4$~$gamma~$cm$^{-2}$s$^{-1}$. The resulting activity in the walls of the cavern can be characterised as $220pm60$~Bq/kg of $^{40}$K, $29pm15$~Bq/kg of $^{238}$U, and $13pm3$~Bq/kg of $^{232}$Th.