We have made measurements of the Sunyaev-Zeldovich (SZ) effect in six galaxy clusters at z > 0.2 using the Sunyaev-Zeldovich Infrared Experiment (SuZIE II) in three frequency bands between 150 and 350 GHz. Simultaneous multi-frequency measurements have been used to distinguish between thermal and kinematic components of the SZ effect, and to significantly reduce the effects of variations in atmospheric emission which can otherwise dominate the noise. We have set limits to the peculiar velocities of each cluster with respect to the Hubble flow, and have used the cluster sample to set a 95% confidence limit of < 1410 km/s to the bulk flow of the intermediate-redshift universe in the direction of the CMB dipole. This is the first time that SZ measurements have been used to constrain bulk flows. We show that systematic uncertainties in peculiar velocity determinations from the SZ effect are likely to be dominated by submillimeter point sources and we discuss the level of this contamination.