We report a comparative polarized Raman study of Weyl semimetals TaAs, NbAs, TaP and NbP. The evolution of the phonon frequencies with the sample composition allows us to determine experimentally which atoms are mainly involved for each vibration mode. Our results confirm previous first-principles calculations indicating that the A$_1$, B$_1(2)$, E$(2)$ and E$(3)$ modes involve mainly the As(P) atoms, the B$_1(1)$ mode is mainly related to Ta(Nb) atoms, and the E$(1)$ mode involves both kinds of atoms. By comparing the energy of the different modes, we establish that the B$_1(1)$, B$_1(2)$, E$(2)$ and E$(3)$ become harder with chemical pressure increasing. This behavior differs from our observation on the A$_1$ mode, which decreases in energy, in contrast to its behavior under external pressure.