We present precise values of the dipole polarizabilities ($alpha$) of the ground $rm [4f^{14}6s] ~ ^2S_{1/2}$ and metastable $rm [4f^{14} 5d] ~ ^2D_{3/2}$ states of Yb$^+$, that are %vital {bf important} in reducing systematics in the clock frequency of the $rm[4f^{14}6s] ~ ^2S_{1/2} rightarrow [4f^{14}5d] ~ ^2D_{3/2}$ transition. The static values of $alpha$ for the ground and $rm [4f^{14} 5d] ~ ^2D_{3/2}$ states are estimated to be $9.8(1) times 10^{-40} ,,rm Jm^2V^{-2}$ and $17.6(5) times 10^{-40},, rm Jm^2V^{-2}$, respectively, while the tensor contribution to the $rm [4f^{14} 5d] ~ ^2D_{3/2}$ state as $- 12.3(3) times 10^{-40},, rm Jm^2V^{-2}$ compared to the experimental value $-13.6(2.2) times 10^{-40},,rm Jm^2V^{-2}$. This corresponds to the differential scalar polarizability value of the above transition as $-7.8$(5)$,times, 10^{-40},rm Jm^2 V^{-2}$ in contrast to the available experimental value $-6.9$(1.4)$,times, 10^{-40}$,, $rm Jm^2V^{-2}$. This results in the black-body radiation (BBR) shift of the clock transition as $-0.44(3)$ Hz at the room temperature, which is large as compared to the previously estimated values. Using the dynamic $alpha$ values, we report the tune-out and magic wavelengths that could be of interest to subdue %major systematics due to the Stark shifts and for constructing lattice optical clock using Yb$^+$.