This article is concerned with stochastic differential equations driven by a $d$ dimensional fractional Brownian motion with Hurst parameter $H>1/4$, understood in the rough paths sense. Whenever the coefficients of the equation satisfy a uniform hypoellipticity condition, we establish a sharp local estimate on the associated control distance function and a sharp local lower estimate on the density of the solution. Our methodology relies heavily on the rough paths structure of the equation.