It is known from both theory and experiments that introducing time delays into the communication network of mobile-agent swarms produces coherent rotational patterns. Often such spatio-temporal rotations can be bistable with other swarming patterns, such as milling and flocking. Yet, most known bifurcation results related to delay-coupled swarms rely on inaccurate mean-field techniques. As a consequence, the utility of applying macroscopic theory as a guide for predicting and controlling swarms of mobile robots has been limited. To overcome this limitation, we perform an exact stability analysis of two primary swarming patterns in a general model with time-delayed interactions. By correctly identifying the relevant spatio-temporal modes that determine stability in the presence of time delay, we are able to accurately predict bistability and unstable oscillations in large swarm simulations-- laying the groundwork for comparisons to robotics experiments.