We present a search for periodic gravitational waves from the neutron star in the supernova remnant Cassiopeia A. The search coherently analyzes data in a 12-day interval taken from the fifth science run of the Laser Interferometer Gravitational-Wave Observatory. It searches gravitational wave frequencies from 100 to 300 Hz, and covers a wide range of first and second frequency derivatives appropriate for the age of the remnant and for different spin-down mechanisms. No gravitational wave signal was detected. Within the range of search frequencies, we set 95% confidence upper limits of 0.7--1.2e-24 on the intrinsic gravitational wave strain, 0.4--4e-4 on the equatorial ellipticity of the neutron star, and 0.005--0.14 on the amplitude of r-mode oscillations of the neutron star. These direct upper limits beat indirect limits derived from energy conservation and enter the range of theoretical predictions involving crystalline exotic matter or runaway r-modes. This is the first gravitational wave search to present upper limits on r-modes.