The observed delay of GRB 170817A relative to GW170817 carries significant information about gamma-ray burst (GRB) physics and is subject to intense debate. In this letter, we put forward an approach to discuss the major source of this time delay. First of all, we use the structured jet model to fit the X-ray/optical/radio afterglows of GRB 170817A together with superluminal motion measured by the Very Long Baseline Interferometry. Our structured jet is modelled with angle-dependent energy and baryon loading. It is found that our model can well fit the afterglows of GRB 170817A. After that, the baryon loading in the jet is inferred based on our fitting results. By comparing the baryon loading to the mass outflow in different stages, we infer that the time lag of the jet launch relative to the merger is less than hundreds or tens of milliseconds. It suggests that the time delay of GRB 170817A relative to GW170817 is defined mostly by the spreading time of the jet propagating to its dissipation radius.