The cohomological rigidity problem for toric manifolds asks whether toric manifolds are diffeomorphic (or homeomorphic) if their integral cohomology rings are isomorphic. Many affirmative partial solutions to the problem have been obtained and no counterexample is known. In this paper, we study the diffeomorphism classification of toric Fano $d$-folds with $d=3,4$ or with Picard number $ge 2d-2$. In particular, we show that those manifolds except for two toric Fano $4$-folds are diffeomorphic if their integral cohomology rings are isomorphic. The exceptional two toric Fano $4$-folds (their ID numbers are 50 and 57 on a list of {O}bro) have isomorphic cohomology rings and their total Pontryagin classes are preserved under an isomorphism between their cohomology rings, but we do not know whether they are diffeomorphic or homeomorphic.