We report on a 63ks Chandra observation of the X-ray transient Swift J195509.6+261406 discovered as the afterglow of what was first believed to be a long duration Gamma-Ray Burst (GRB 070610). The outburst of this source was characterized by unique optical flares on timescales of second or less, morphologically similar to the short X-ray bursts usually observed from magnetars. Our Chandra observation was performed ~2 years after the discovery of the optical and X-ray flaring activity of this source, catching it in its quiescent state. We derive stringent upper limits on the quiescent emission of Swif J195509.6+261406 which argues against the possibility of this object being a typical magnetar. Our limits show that the most viable interpretation on the nature of this peculiar bursting source, is a binary system hosting a black hole or a neutron star with a low mass companion star (< 0.12 M_{odot}), and with an orbital period smaller than a few hours.