$In$ $situ$ angle-resolved photoemission spectroscopy (ARPES) has been performed on SrVO$_3$ ultrathin films, which show metallic quantum well (QW) states, to unveil the origin of the anomalous mass enhancement in the QW subbands. The line-shape analysis of the ARPES spectra reveals that the strength of the electron correlation increases as the subband bottom energy approaches the Fermi level. These results indicate that the anomalous subband-dependent mass enhancement mainly arises from the quasi-one-dimensional character of confined V $3d$ states as a result of their orbital-selective quantization.