Cosmic rays (CRs) are tracers of solar events when they are associated with solar flares, but also galactic events when they come from outside our solar system. SEPs are correlated with the 11-year solar cycle while GCRs are anti-correlated due to their interaction with the heliospheric magnetic field and the solar wind. Our aim is to quantify separately the impact of the amplitude and the geometry of the magnetic field on the propagation of CRs of various energies in the inner heliosphere. We focus especially on the diffusion caused by the magnetic field along and across the field lines. To do so, we use the results of 3D MHD wind simulations running from the lower corona up to 1 AU. The wind is modeled using a polytropic approximation, and fits and power laws are used to account for the turbulence. Using these results, we compute the parallel and perpendicular diffusion coefficients of the Parker CR transport equation, yielding 3D maps of the diffusion of CRs in the inner heliosphere. By varying the amplitude of the magnetic field, we change the amplitude of the diffusion by the same factor, and the radial gradients by changing the spread of the current sheet. By varying the geometry of the magnetic field, we change the latitudinal gradients of diffusion by changing the position of the current sheets. By varying the energy, we show that the distribution of values for SEPs is more peaked than GCRs. For realistic solar configurations, we show that diffusion is highly non-axisymmetric due to the configuration of the current sheets, and that the distribution varies a lot with the distance to the Sun with a drift of the peak value. This study shows that numerical simulations and theory can help quantify better the influence of the various magnetic field parameters on the propagation of CRs. This study is a first step towards generating synthetic CR rates from numerical simulations.