Determining the average fraction of Lyman continuum (LyC) photons escaping high redshift galaxies is essential for understanding how reionization proceeded in the z>6 Universe. We want to measure the LyC signal from a sample of sources in the Chandra Deep Field South (CDFS) and COSMOS fields for which ultra-deep VIMOS spectroscopy as well as multi-wavelength Hubble Space Telescope (HST) imaging are available. We select a sample of 46 galaxies at $zsim 4$ from the VIMOS Ultra Deep Survey (VUDS) database, such that the VUDS spectra contain the LyC part of the spectra, that is, the rest-frame range $880-910AA$. Taking advantage of the HST imaging, we apply a careful cleaning procedure and reject all the sources showing nearby clumps with different colours, that could potentially be lower-redshift interlopers. After this procedure, the sample is reduced to 33 galaxies. We measure the ratio between ionizing flux (LyC at $895AA$) and non-ionizing emission (at $sim 1500 AA$) for all individual sources. We also produce a normalized stacked spectrum of all sources. Assuming an intrinsic average $L_{ u}(1470)/L_{ u}(895)$ of 3, we estimate the individual and average relative escape fraction. We do not detect ionizing radiation from any individual source, although we identify a possible LyC emitter with very high Ly$alpha$ equivalent width (EW). From the stacked spectrum and assuming a mean transmissivity for the sample, we measure a relative escape fraction $f_{esc}^{rel}=0.09pm0.04$. We also look for correlations between the limits in the LyC flux and source properties and find a tentative correlation between LyC flux and the EW of the Ly$alpha$ emission line. Our results imply that the LyC flux emitted by $V=25-26$ star-forming galaxies at z$sim$4 is at most very modest, in agreement with previous upper limits from studies based on broad and narrow band imaging.