We present realistic simulations of quantum confinement effects in ballistic graphene quantum dots with linear dimensions of 10 to 40 nm. We determine wavefunctions and energy level statistics in the presence of disorder resulting from edge roughness, charge impurities, or short-ranged scatterers. Marked deviations from a simple Dirac billiard for massless fermions are found. We find a remarkably stable dependence of the nearest-neighbor level spacing on edge roughness suggesting that the roughness of fabricated devices can be potentially characterized by the distribution of measured Coulomb blockade peaks.