By studying Rozansky-Witten theory with non-compact target spaces we find new connections with knot invariants whose physical interpretation was not known. This opens up several new avenues, which include a new formulation of $q$-series invariants of 3-manifolds in terms of affine Grassmannians and a generalization of Akutsu-Deguchi-Ohtsuki knot invariants.