Large collections of coupled, heterogeneous agents can manifest complex dynamical behavior presenting difficulties for simulation and analysis. However, if the collective dynamics lie on a low-dimensional manifold then the original agent-based model may be approximated with a simplified surrogate model on and near the low-dimensional space where the dynamics live. This is typically accomplished by deriving coarse variables that summarize the collective dynamics, these may take the form of either a collection of scalars or continuous fields (e.g. densities), which are then used as part of a reduced model. Analytically identifying such simplified models is challenging and has traditionally been accomplished through the use of mean-field reductions or an Ott-Antonsen ansatz, but is often impossible. Here we present a data-driven coarse-graining methodology for discovering such reduced models. We consider two types of reduced models: globally-based models which use global information and predict dynamics using information from the whole ensemble, and locally-based models that use local information, that is, information from just a subset of agents close (close in heterogeneity space, not physical space) to an agent, to predict the dynamics of an agent. For both approaches we are able to learn laws governing the behavior of the reduced system on the low-dimensional manifold directly from time series of states from the agent-based system. A nontrivial conclusion is that the dynamics can be equally well reproduced by an all-to-all coupled as well as by a locally coupled model of the same agents.