We study how the first galaxies were assembled under feedback from the accretion onto a central black hole (BH) that is left behind by the first generation of metal-free stars through self-consistent, cosmological simulations. X-ray radiation from the accretion of gas onto BH remnants of Population III (Pop III) stars, or from high-mass X-ray binaries (HMXBs), again involving Pop III stars, influences the mode of second generation star formation. We track the evolution of the black hole accretion rate and the associated X-ray feedback starting with the death of the Pop III progenitor star inside a minihalo and following the subsequent evolution of the black hole as the minihalo grows to become an atomically cooling galaxy. We find that X-ray photoionization heating from a stellar-mass BH is able to quench further star formation in the host halo at all times before the halo enters the atomic cooling phase. X-ray radiation from a HMXB, assuming a luminosity close to the Eddington value, exerts an even stronger, and more diverse, feedback on star formation. It photoheats the gas inside the host halo, but also promotes the formation of molecular hydrogen and cooling of gas in the intergalactic medium and in nearby minihalos, leading to a net increase in the number of stars formed at early times. Our simulations further show that the radiative feedback from the first BHs may strongly suppress early BH growth, thus constraining models for the formation of supermassive BHs.