Finding the ground state of a fermionic Hamiltonian using quantum Monte Carlo is a very difficult problem, due to the Fermi sign problem. While still scaling exponentially, full configuration-interaction Monte Carlo (FCI-QMC) mitigates some of the exponential variance by allowing annihilation of noise -- whenever two walkers arrive at the same configuration with opposite signs, they are removed from the simulation. While FCI-QMC has been quite successful for quantum chemistry problems, its application to problems in condensed systems has been limited. In this paper, we apply FCI-QMC to the Fermi polaron problem, which provides an ideal test-bed for improving the algorithm. In its simplest form, FCI-QMC is unstable for even a fairly small system sizes. However, with a series of algorithmic improvements, we are able to significantly increase its effectiveness. We modify fixed node QMC to work in these systems, introduce a well chosen importance sampled trial wave function, a partial node approximation, and a variant of released node. Finally, we develop a way to perform FCI-QMC directly in the thermodynamic limit.