The unprecedented wide bandgap tunability (~1 eV) of Al$_x$In$_{1-x}$As$_y$Sb$_{1-y}$ latticed-matched to GaSb enables the fabrication of photodetectors over a wide range from near-infrared to mid-infrared. In this paper, the valence band-offsets in AlxIn1-xAsySb1-y with different Al compositions are analyzed by tight-binding calculations and X-ray photoelectron spectroscopy (XPS) measurements. The observed weak variation in valence band offsets is consistent with the lack of any minigaps in the valence band, compared to the conduction band.