Let $K subset R^d$ be a smooth convex set and let $P_la$ be a Poisson point process on $R^d$ of intensity $la$. The convex hull of $P_la cap K$ is a random convex polytope $K_la$. As $la to infty$, we show that the variance of the number of $k$-dimensional faces of $K_la$, when properly scaled, converges to a scalar multiple of the affine surface area of $K$. Similar asymptotics hold for the variance of the number of $k$-dimensional faces for the convex hull of a binomial process in $K$.