We theoretically compute the thermal conductivity of SiGe alloy nanowires as a function of nanowire diameter, alloy concentration, and temperature, obtaining a satisfactory quantitative agreement with experimental results. Our results account for the weaker diameter dependence of the thermal conductivity recently observed in Si$_{1-x}$Ge$_x$ nanowires ($x<0.1$), as compared to pure Si nanowires. We also present calculations in the full range of alloy concentrations, $0 leq x leq 1$, which may serve as a basis for comparison with future experiments on high alloy concentration nanowires.