We introduce The Novel Probes Project, an initiative to advance the field of astrophysical tests of the dark sector by creating a forum that connects observers and theorists. This review focuses on tests of gravity and is intended to be of use primarily to observers, but also to theorists with interest in the development of experimental tests. It is twinned with a separate review on tests of dark matter self-interactions (Adhikari et al., in prep.). Our focus is on astrophysical probes of gravity in the weak-field regime, ranging from stars to quasilinear cosmological scales. These are complementary to both strong-field tests and background and linear probes in cosmology. In particular, the nonlinear screening mechanisms that are an integral part of viable modified gravity models lead to characteristic signals specifically on astrophysical scales. The constraining power of these signals is not limited by cosmic variance, but comes with the challenge of building robust theoretical models of the nonlinear dynamics of stars, galaxies, clusters and large scale structure. In this review we lay the groundwork for a thorough exploration of the astrophysical regime with an eye to using the current and next generation of observations for tests of gravity. We begin by setting the scene for how theories beyond General Relativity are expected to behave, focusing primarily on screened fifth forces. We describe the analytic and numerical techniques for exploring the pertinent astrophysical systems, as well as the signatures of modified gravity. With these in hand we present a range of observational tests, and discuss prospects for future measurements and theoretical developments.