Radii of charge and neutron distributions are fundamental nuclear properties. They depend on both nuclear interaction parameters related to the equation of state of infinite nuclear matter and on quantal shell effects, which are strongly impacted by the presence of nuclear surface. In this work, by studying the dependence of charge and neutron radii, and neutron skin, on nuclear matter parameters, we assess different mechanisms that drive nuclear sizes. We apply nuclear density functional theory using a family of Skyrme functionals obtained by means of different optimization protocols targeting specific nuclear properties. By performing the Monte-Carlo sampling of reasonable functionals around the optimal parametrization, we study correlations between nuclear matter paramaters and observables characterizing charge and neutron distributions. We demonstrate the existence of the strong converse relation between the nuclear charge radii and the saturation density of symmetric nuclear matter and also between the neutron skins and the slope of the symmetry energy. For functionals optimized to experimental binding energies only, proton and neutron radii are weakly correlated due to canceling trends from different nuclear matter parameters. We show that by requiring that the nuclear functional reproduces the empirical saturation point of symmetric nuclear matter practically fixes the charge (or proton) radii, and vice versa. The neutron skin uncertainty primarily depends on the slope of the symmetry energy. Consequently, imposing a constraint on both $rho_0$ and $L$ practically determines the nuclear size, modulo small variations due to shell effects.