Studying the interplay between superconductivity and quantum magnetotransport in two-dimensional materials has been a topic of interest in recent years. Towards such a goal it is important to understand the impact of magnetic field on the charge transport at the superconductor-normal channel (SN) interface. Here we carried out a comprehensive study of Andreev conductance under weak magnetic fields using diffusive superconductor- graphene Josephson weak links. We observe that the Andreev conductance is suppressed even in magnetic fields far below the upper critical field of the superconductor. The suppression of Andreev conductance depends on and can be minimized by controlling the ramping of the magnetic field. We identify that the key factor behind this suppression is the reduction of the superconducting gap due to the piling of vortices on the superconducting contacts. In devices where superconducting gap at the superconductor-graphene interface is heavily reduced by proximity effect, the enlarged vortex cores overlap quickly with increasing magnetic field, resulting in a rapid decrease of the interfacial gap. However, in weak links with relatively large effective superconducting gap the AR conductance persists up to the upper critical field. Our results provide guidance to the study of quantum material-superconductor systems in presence of magnetic field, where survival of induced superconductivity is critical.