Unlike hot Jupiters or other gas giants, super-Earths are expected to have a wide variety of compositions, ranging from terrestrial bodies like our own to more gaseous planets like Neptune. Observations of transiting systems, which allow us to directly measure planet masses and radii and constrain atmospheric properties, are key to understanding the compositional diversity of the planets in this mass range. Although Kepler has discovered hundreds of transiting super-Earth candidates over the past four years, the majority of these planets orbit stars that are too far away and too faint to allow for detailed atmospheric characterization and reliable mass estimates. Ground-based transit surveys focus on much brighter stars, but most lack the sensitivity to detect planets in this size range. One way to get around the difficulty of finding these smaller planets in transit is to start by choosing targets that are already known to contain super-Earth sized bodies detected using the radial velocity technique. Here we present results from a Spitzer program to observe six of the most favorable RV detected super-Earth systems, including HD 1461, HD 7924, HD 156668, HIP 57274, and GJ 876. We find no evidence for transits in any of their 4.5 micron flux light curves, and place limits on the allowed transit depths and corresponding planet radii that rule out even the most dense and iron-rich compositions for these objects. We also observed HD 97658, but the observation window was based on a possible ground-based transit detection (Henry et al. 2011) that was later ruled out; thus the window did not include the predicted time for the transit detection recently made by MOST (Dragomir et al. 2013).