The nuclear symmetry energy is a key quantity in nuclear (astro)physics. It describes the isospin dependence of the nuclear equation of state (EOS), which is commonly assumed to be almost quadratic. In this work, we confront this standard quadratic expansion of the EOS with explicit asymmetric nuclear-matter calculations based on a set of commonly used Hamiltonians including two- and three-nucleon forces derived from chiral effective field theory. We study, in particular, the importance of non-quadratic contributions to the symmetry energy, including the non-analytic logarithmic term introduced by Kaiser [Phys.~Rev.~C textbf{91}, 065201 (2015)]. Our results suggest that the quartic contribution to the symmetry energy can be robustly determined from the various Hamiltonians employed, and we obtain 1.00(8) MeV (or 0.55(8) MeV for the potential part) at saturation density, while the logarithmic contribution to the symmetry energy is relatively small and model-dependent. We finally employ the meta-model approach to study the impact of the higher-order contributions on the neutron-star crust-core transition density, and find a small 5% correction.