We report the first detection of asymmetry in a supernova (SN) photosphere based on SN light echo (LE) spectra of Cas A from the different perspectives of dust concentrations on its LE ellipsoid. New LEs are reported based on difference images, and optical spectra of these LEs are analyzed and compared. After properly accounting for the effects of finite dust-filament extent and inclination, we find one field where the He I and H alpha features are blueshifted by an additional ~4000 km/s relative to other spectra and to the spectra of the Type IIb SN 1993J. That same direction does not show any shift relative to other Cas A LE spectra in the Ca II near-infrared triplet feature. We compare the perspectives of the Cas A LE dust concentrations with recent three-dimensional modeling of the SN remnant (SNR) and note that the location having the blueshifted He I and H alpha features is roughly in the direction of an Fe-rich outflow and in the opposite direction of the motion of the compact object at the center of the SNR. We conclude that Cas A was an intrinsically asymmetric SN. Future LE spectroscopy of this object, and of other historical SNe, will provide additional insight into the connection of explosion mechanism to SN to SNR, as well as give crucial observational evidence regarding how stars explode.