Polarization dependent vanadium L edge X-ray absorption spectra of BaVS$_3$ single crystals are measured in the four phases of the compound. The difference between signals with the polarization textbf{E}$perp$textbf{c} and textbf{E}$parallel$textbf{c
} (linear dichroism) changes with temperature. Besides increasing intensity of one of the maxima, a new structure appears in the pre-edge region below the metal-insulator transition. More careful examination brings to light that the changes start already with pretransitional charge density wave fluctuations. Simple symmetry analysis suggests that the effect is related to rearrangements in $E_{g}$ and $A_{1g}$ states, and is compatible with the formation of four inequivalent V sites along the V-S chain.
