Semiconductor quantum dots in cavities are promising single-photon sources. Here, we present a path to deterministic operation, by harnessing the intrinsic linear dipole in a neutral quantum dot via phonon-assisted excitation. This enables emission of fully polarized single photons, with a measured degree of linear polarization up to 0.994 $pm$ 0.007, and high population inversion -- 85% as high as resonant excitation. We demonstrate a single-photon source with a polarized first lens brightness of 0.51 $pm $ 0.01, a single-photon purity of 0.939 $pm$ 0.001 and single-photon indistinguishability of 0.915 $pm$ 0.003.