We reconsider the thermal scalar Casimir effect for $p$-dimensional rectangular cavity inside $D+1$-dimensional Minkowski space-time. We derive rigorously the regularization of the temperature-dependent part of the free energy by making use of the Abel-Plana formula repeatedly and get the explicit expression of the terms to be subtracted. In the cases of $D$=3, $p$=1 and $D$=3, $p$=3, we precisely recover the results of parallel plates and three-dimensional box in the literature. Furthermore, for $D>p$ and $D=p$ cases with periodic, Dirichlet and Neumann boundary conditions, we give the explicit expressions of the Casimir free energy in both low temperature (small separations) and high temperature (large separations) regimes, through which the asymptotic behavior of the free energy changing with temperature and the side length is easy to see. We find that for $D>p$, with the side length going to infinity, the Casimir free energy tends to positive or negative constants or zero, depending on the boundary conditions. But for $D=p$, the leading term of the Casimir free energy for all three boundary conditions is a logarithmic function of the side length. We also discuss the thermal Casimir force changing with temperature and the side length in different cases and find with the side length going to infinity the force always tends to zero for different boundary conditions regardless of $D>p$ or $D=p$. The Casimir free energy and force at high temperature limit behave asymptotically alike in that they are proportional to the temperature, be they positive (repulsive) or negative (attractive) in different cases. Our study may be helpful in providing a comprehensive and complete understanding of this old problem.