We present results for the isovector electromagnetic form factors of the nucleon computed on the CLS ensembles with $N_f=2+1$ flavors of $mathcal{O}(a)$-improved Wilson fermions and an $mathcal{O}(a)$-improved vector current. The analysis includes ensembles with four lattice spacings and pion masses ranging from 130 MeV up to 350 MeV and mainly targets the low-$Q^2$ region. In order to remove any bias from unsuppressed excited-state contributions, we investigate several source-sink separations between 1.0 fm and 1.5 fm and apply the summation method as well as explicit two-state fits. The chiral interpolation is performed by applying covariant chiral perturbation theory including vector mesons directly to our form factor data, thus avoiding an auxiliary parametrization of the $Q^2$ dependence. At the physical point, we obtain $mu=4.71(11)_{mathrm{stat}}(13)_{mathrm{sys}}$ for the nucleon isovector magnetic moment, in good agreement with the experimental value and $langle r_mathrm{M}^2rangle~=~0.661(30)_{mathrm{stat}}(11)_{mathrm{sys}},~mathrm{fm}^2$ for the corresponding square-radius, again in good agreement with the value inferred from the $ep$-scattering determination [Bernauer et~al., Phys. Rev. Lett., 105, 242001 (2010)] of the proton radius. Our estimate for the isovector electric charge radius, $langle r_mathrm{E}^2rangle = 0.800(25)_{mathrm{stat}}(22)_{mathrm{sys}},~mathrm{fm}^2$, however, is in slight tension with the larger value inferred from the aforementioned $ep$-scattering data, while being in agreement with the value derived from the 2018 CODATA average for the proton charge radius.