We classify finite $p$-groups, upto isoclinism, which have only two conjugacy class sizes $1$ and $p^3$. It turns out that the nilpotency class of such groups is $2$.