We present photometric redshifts for an uniquely large and deep sample of 522286 objects with i_{AB}<25 in the Canada-France Legacy Survey ``Deep Survey fields, which cover a total effective area of 3.2 deg^2. We use 3241 spectroscopic redshifts with 0<z<5 from the VIMOS VLT Deep Survey as a calibration to derive these photometric redshifts. We devise a robust calibration method which removes systematic trends in the photometric redshifts and significantly reduces the fraction of catastrophic errors. We use our unique spectroscopic sample to present a detailed assessment of the robustness of the photometric redshift sample. For a sample selected at i_{AB}<24, we reach a redshift accuracy of sigma_{Delta z/(1+z)}=0.037 with eta=3.7% of catastrophic error. The reliability of our photometric redshifts is lower for fainter objects: we find sigma_{Delta z/(1+z)}=0.029, 0.043 and eta=1.7%, 5.4% for samples selected at i_{AB}=17.5-22.5 and 22.5-24 respectively. We find that the photometric redshifts of starburst galaxies in our sample are less reliable: although these galaxies represent only 18% of the spectroscopic sample they are responsible for 54% of the catastrophic errors. We find an excellent agreement between the photometric and the VVDS spectroscopic redshift distributions at i_{AB}<24. Finally, we compare the redshift distributions of i selected galaxies on the four CFHTLS deep fields, showing that cosmic variance is already present on fields of 0.8 deg^2.