The symmetry energy and its density dependence are pivotal for many nuclear physics and astrophysics applications, as they determine properties ranging from the neutron-skin thickness of nuclei to the crust thickness and the radius of neutron stars. Recently, PREX-II reported a value of $0.283pm0.071$ fm for the neutron-skin thickness of $^{208}$Pb, $R_{rm skin}^{^{208}text{Pb}}$, implying a symmetry-energy slope parameter $L$ of $106pm37$ MeV, larger than most ranges obtained from microscopic calculations and other nuclear experiments. We use a nonparametric equation of state representation based on Gaussian processes to constrain the symmetry energy $S_0$, $L$, and $R_{rm skin}^{^{208}text{Pb}}$ directly from observations of neutron stars with minimal modeling assumptions. The resulting astrophysical constraints from heavy pulsar masses, LIGO/Virgo, and NICER favor smaller values of the neutron skin and $L$, as well as negative symmetry incompressibilities. Combining astrophysical data with chiral effective field theory ($chi$EFT) and PREX-II constraints yields $S_0 = 33.0^{+2.0}_{-1.8}$ MeV, $L=53^{+13}_{-15}$ MeV, and $R_{rm skin}^{^{208}text{Pb}} = 0.17^{+0.04}_{-0.04}$ fm. We also examine the consistency of several individual $chi$EFT calculations with astrophysical observations and terrestrial experiments. We find that there is only mild tension between $chi$EFT, astrophysical data, and PREX-IIs $R_mathrm{skin}^{^{208}mathrm{Pb}}$ measurement ($p$-value $= 12.3%$) and that there is excellent agreement between $chi$EFT, astrophysical data, and other nuclear experiments.