For a partition $lambda$ of $n in {mathbb N}$, let $I^{rm Sp}_lambda$ be the ideal of $R=K[x_1,ldots,x_n]$ generated by all Specht polynomials of shape $lambda$. In the previous paper, the second author showed that if $R/I^{rm Sp}_lambda$ is Cohen-Macaulay, then $lambda$ is either $(n-d,1,ldots,1),(n-d,d)$, or $(d,d,1)$, and the converse is true if ${rm char}(K)=0$. In this paper, we compute the Hilbert series of $R/I^{rm Sp}_lambda$ for $lambda=(n-d,d)$ or $(d,d,1)$. Hence, we get the Castelnuovo-Mumford regularity of $R/I^{rm Sp}_lambda$, when it is Cohen-Macaulay. In particular, $I^{rm Sp}_{(d,d,1)}$ has a $(d+2)$-linear resolution in the Cohen-Macaulay case.