In this brief note, we determine the fluctuations of the initial eccentricity in heavy-ion collisions caused by fluctuations of the nucleon configurations. This is done via a Monte-Carlo implementation of a Color Glass Condensate $k_t$-factorization
approach. The eccentricity fluctuations are found to nearly saturate elliptic flow fluctuations measured recently at RHIC. Extrapolations to LHC energies are shown.
