Possible connections between central black-hole (BH) growth and host-galaxy compactness have been found observationally, which may provide insight into BH-galaxy coevolution: compact galaxies might have large amounts of gas in their centers due to their high mass-to-size ratios, and simulations predict that high central gas density can boost BH accretion. However, it is not yet clear if BH growth is fundamentally related to the compactness of the host galaxy, due to observational degeneracies between compactness, stellar mass ($M_bigstar$), and star formation rate (SFR). To break these degeneracies, we carry out systematic partial-correlation studies to investigate the dependence of sample-averaged BH accretion rate ($rm overline{BHAR}$) on the compactness of host galaxies, represented by the surface-mass density, $Sigma_rm e$, or the projected central surface-mass density within 1 kpc, $Sigma_1$. We utilize 8842 galaxies with H < 24.5 in the five CANDELS fields at z = 0.5-3. We find that $rm overline{BHAR}$ does not significantly depend on compactness when controlling for SFR or $M_bigstar$ among bulge-dominated galaxies and galaxies that are not dominated by bulges, respectively. However, when testing is confined to star-forming galaxies at z = 0.5-1.5, we find that the $rm overline{BHAR}$-$Sigma_1$ relation is not simply a secondary manifestation of a primary $rm overline{BHAR}$-$M_bigstar$ relation, which may indicate a link between BH growth and the gas density within the central 1 kpc of galaxies.