The thermal expansion of the heavy fermion compound CeRh2Si2 has been measured under pressure as a function of temperature using strain gages. A large anomaly associated to the Neel temperature has been detected even above the suspected critical pressure Pc = 1.05 GPa where no indication of antiferromagnetism has been observed in calorimetry experiments sensitive to the entropy change. An unexpected feature is the pressure slowdown of the antiferromagnetic-paramagnetic transition by comparison to the fast pressure collapse predicted for homogeneous first order quantum phase transition with one unique pressure singularity at Pc. A large pressure dependance is observed in the anisotropy of the thermal expansion measured parallel or perpendicular to the c axis of this tetragonal crystal. The Fermi surface reconstruction associated to the first order transition produces quite different pressure response in the transport scattering measured along different crystallographic directions. A brief discussion is made on other examples of first order quantum transitions in strongly correlated electronic systems : MnSi and CeCoIn5.