Photonic platforms are an excellent setting for quantum technologies because weak photon-environment coupling ensures long coherence times. The second key ingredient for quantum photonics is interactions between photons, which can be provided by optical nonlinearities in the form of cross-phase-modulation (XPM). This approach underpins many proposed applications in quantum optics and information processing, but achieving its potential requires strong single-photon-level nonlinear phase shifts and also scalable nonlinear elements. In this work we show that the required nonlinearity can be provided by exciton-polaritons in micropillars with embedded quantum wells. These combine the strong interactions of excitons with the scalability of micrometer-sized emitters. We observe XPM up to $3 pm 1$ mrad per particle using laser beams attenuated to below single photon average intensity. With our work serving as a first stepping stone, we lay down a route for quantum information processing in polaritonic lattices.