A spectral convex set is a collection of symmetric matrices whose range of eigenvalues form a symmetric convex set. Spectral convex sets generalize the Schur-Horn orbitopes studied by Sanyal-Sottile-Sturmfels (2011). We study this class of convex bodies, which is closed under intersections, polarity, and Minkowski sums. We describe orbits of faces and give a formula for their Steiner polynomials. We then focus on spectral polyhedra. We prove that spectral polyhedra are spectrahedra and give small representations as spectrahedral shadows. We close with observations and questions regarding hyperbolicity cones, polar convex bodies, and spectral zonotopes.