Since the current evidence of its existence is revealed only through its gravitational influence, the way dark matter couples to gravity must be then of primary importance. Here, unlike the standard model sector which is typically coupled to metric, dark matter is supposed to couple only to spacetime affine connection through a $Z_2$-symmetry breaking term. We show that this structure leads to a coupling between dark matter, which is considered scalar, and the standard model Higgs potential. This induces dark matter decays into standard model particles through the Higgs which acts as a portal between the visible and the dark sectors. We study thoroughly the resulting decay modes for various mass ranges, and provide relevant bounds on the nonminimal coupling to affine gravity in line with observational data. Moreover, we find that the coupling to Higgs can be sufficiently large to facilitate production of dark matter lighter than 10 GeV at current and future high energy colliders.