We present spectropolarimetric observations of the nearby Type Ia SN 2014J in M82 over six epochs: +0, +7, +23, +51, +77, +109, and +111 days with respect to B-band maximum. The strong continuum polarization, which is constant with time, shows a wavelength dependence unlike that produced by linear dichroism in Milky Way dust. The observed polarization may be due entirely to interstellar dust or include a circumstellar scattering component. We find that the polarization angle aligns with the magnetic field of the host galaxy, arguing for an interstellar origin. Additionally, we confirm a peak in polarization at short wavelengths that would imply $R_V < 2 $ along the light of sight, in agreement with earlier polarization measurements. For illustrative purposes, we include a two component fit to the continuum polarization of our +51 day epoch that combines a circumstellar scattering component with interstellar dust where scattering can account for over half of the polarization at $4000$ AA. Upon removal of the interstellar polarization signal, SN 2014J exhibits very low levels of continuum polarization. Asymmetries in the distribution of elements within the ejecta are visible through moderate levels of time-variable polarization in accordance with the Si II 6355 AA absorption line. At maximum light, the line polarization reaches $sim0.6$% and decreases to $sim0.4%$ one week later. This feature also forms a loop on the $q_{RSP}$-$u_{RSP}$ plane illustrating that the ion does not have an axisymmetric distribution. The observed polarization properties suggest the explosion geometry of SN 2014J is generally spheroidal with a clumpy distribution of silicon.