We use deep Chandra X-ray imaging to measure the distribution of specific black hole accretion rates ($L_X$ relative to the stellar mass of the galaxy) and thus trace AGN activity within star-forming and quiescent galaxies, as a function of stellar mass (from $10^{8.5}-10^{11.5} M_odot$) and redshift (to $z sim 4$). We adopt near-infrared selected samples of galaxies from the CANDELS and UltraVISTA surveys, extract X-ray data for every galaxy, and use a flexible Bayesian method to combine these data and to measure the probability distribution function of specific black hole accretion rates, $lambda_{sBHAR}$. We identify a broad distribution of $lambda_{sBHAR}$ in both star-forming and quiescent galaxies---likely reflecting the stochastic nature of AGN fuelling---with a roughly power-law shape that rises toward lower $lambda_{sBHAR}$, a steep cutoff at $lambda_{sBHAR} gtrsim 0.1-1$ (in Eddington equivalent units), and a turnover or flattening at $lambda_{sBHAR} lesssim 10^{-3}-10^{-2}$. We find that the probability of a star-forming galaxy hosting a moderate $lambda_{sBHAR}$ AGN depends on stellar mass and evolves with redshift, shifting toward higher $lambda_{sBHAR}$ at higher redshifts. This evolution is truncated at a point corresponding to the Eddington limit, indicating black holes may self-regulate their growth at high redshifts when copious gas is available. The probability of a quiescent galaxy hosting an AGN is generally lower than that of a star-forming galaxy, shows signs of suppression at the highest stellar masses, and evolves strongly with redshift. The AGN duty cycle in high-redshift ($zgtrsim2$) quiescent galaxies thus reaches $sim$20 per cent, comparable to the duty cycle in star-forming galaxies of equivalent stellar mass and redshift.