Metallic oxide SrVO3 represents a prototype system for the study of the mechanism behind thickness-induced metal-to-insulator transition (MIT) or crossover in thin films due to its simple cubic symmetry with one electron in the 3d state in the bulk. Here we report a deviation of chemical composition and distortion of lattice structure existing in the initial 3 unit cells of SrVO3 films grown on SrTiO3 (001) from its bulk form, which shows a direct correlation to the thickness-dependent MIT. In-situ photoemission and scanning tunneling spectroscopy indicate a MIT at the critical thickness of ~3 unit cell (u.c.), which coincides with the formation of a (root2Xroot2)R45 surface reconstruction. However, atomically resolved scanning transmission electron microscopy and electron energy loss spectroscopy show depletion of Sr, change of V valence, thus implying the existence of a significant amount of oxygen vacancies in the 3 u.c. of SrVO3 near the interface. Transport and magneto-transport measurements further reveal that disorder, rather than electron correlations, is likely to be the main cause for the MIT in the SrVO3 ultrathin films.