A pixel detector with a CVD diamond sensor has been studied in a 180 GeV/c pion beam. The charge collection properties of the diamond sensor were studied as a function of the track position, which was measured with a silicon microstrip telescope. Non-uniformities were observed on a length scale comparable to the diamond crystallites size. In some regions of the sensor, the charge drift appears to have a component parallel to the sensor surface (i.e., normal to the applied electric field) resulting in systematic residuals between the track position and the hits position as large as 40 $mu$m. A numerical simulation of the charge drift in polycrystalline diamond was developed to compute the signal induced on the electrodes by the electrons and holes released by the passing particles. The simulation takes into account the crystallite structure, non-uniform trapping across the sensor, diffusion and polarization effects. It is in qualitative agreement with the data. Additional lateral electric field components result from the non-uniform trapping of charges in the bulk. These provide a good explanation for the large residuals observed.