We elucidate the fate of neighboring two and three-$alpha$ particles in cold neutron matter by focusing on an analogy between such $alpha$ systems and Fermi polarons realized in ultracold atoms. We describe in-medium excitation properties of an $alpha$ particle and neutron-mediated two- and three-$alpha$ interactions using theoretical approaches developed for studies of cold atomic systems. We numerically solve the few-body Schrodinger equation of $alpha$ particles within standard $alpha$ cluster models combined with in-medium properties of $alpha$ particles. We point out that the resultant two-$alpha$ ground state and three-$alpha$ first excited state, which correspond to $^8$Be and the Hoyle state, respectively, known as main components in the triple-$alpha$ reaction, can become bound states in such a many-neutron background although these states are unstable in vacuum. Our results suggest a significance of these in-medium cluster states not only in astrophysical environments such as core-collapsed supernova explosions and neutron star mergers but also in neutron-rich nuclei.