Neutron star observations, including direct mass and radius measurements as well as the analysis of gravitational wave signals emitted by stellar mergers, provide valuable and unique insights into the properties of strongly interacting matter at high densities. In this proceedings contribution, I review recent efforts to systematically constrain the equation of state (EoS) of dense nuclear and quark matter using a combination of ab initio particle and nuclear physics calculations and astrophysical data. In particular, I discuss the constraints that the gravitational wave observation GW170817 has placed on the EoS, and comment on the future prospects of improving the accuracy, to which this quantity is known.