Near-field radiative heat transfer between bodies at the nanoscale can surpass blackbody limits on thermal radiation by orders of magnitude due to contributions from evanescent electromagnetic fields, which carry no energy to the far-field. Thus far, principles guiding explorations of larger heat transfer beyond planar structures have assumed utility in surface nanostructuring, which can enhance the density of states, and further assumed that such design paradigms can approach Landauer limits, in analogy to conduction. We derive fundamental shape-independent limits to radiative heat transfer, applicable in near- through far-field regimes, that incorporate material and geometric constraints such as intrinsic dissipation and finite object sizes, and show that these preclude reaching the Landauer limits in all but a few restrictive scenarios. Additionally, we show that the interplay of material response and electromagnetic scattering among proximate bodies means that bodies which maximize radiative heat transfer actually maximize scattering rather than absorption. Finally, we compare our new bounds to existing Landauer limits, as well as limits involving bodies maximizing far-field absorption, and show that these lead to overly optimistic predictions. Our results have ramifications for the ultimate performance of thermophotovoltaics and nanoscale cooling, as well as related incandescent and luminescent devices.