Plasma-based accelerators (PBAs), having demonstrated the production of GeV electron beams in only centimetre scales, offer a path towards a new generation of highly compact and cost-effective particle accelerators. However, achieving the required beam quality, particularly on the energy spread for applications such as free-electron lasers, remains a challenge. Here we investigate fundamental sources of energy spread and bunch length in PBAs which arise from the betatron motion of beam electrons. We present an analytical theory, validated against particle-in-cell simulations, which accurately describes these phenomena. Significant impact on the beam quality is predicted for certain configurations, explaining previously observed limitations on the achievable bunch length and energy spread. Guidelines for mitigating these contributions towards high-quality beams are deduced.