Given a quantum many-body system with few-body interactions, how rapidly can quantum information be hidden during time evolution? The fast scrambling conjecture is that the time to thoroughly mix information among N degrees of freedom grows at least logarithmically in N. We derive this inequality for generic quantum systems at infinite temperature, bounding the scrambling time by a finite decay time of local quantum correlations at late times. Using Lieb-Robinson bounds, generalized Sachdev-Ye-Kitaev models, and random unitary circuits, we propose that a logarithmic scrambling time can be achieved in most quantum systems with sparse connectivity. These models also elucidate how quantum chaos is not universally related to scrambling: we construct random few-body circuits with infinite Lyapunov exponent but logarithmic scrambling time. We discuss analogies between quantum models on graphs and quantum black holes, and suggest methods to experimentally study scrambling with as many as 100 sparsely-connected quantum degrees of freedom.