Kondo insulators are predicted to undergo an insulator-to-metal transition under applied magnetic field, yet the extremely high fields required to date have prohibited a comprehensive investigation of the nature of this transition. Here we show that Ce3Bi4Pd3 provides an ideal platform for this investigation, owing to the unusually small magnetic field of B ~ 11 T required to overcome its Kondo insulating gap. Above Bc, we find a magnetic field-induced Fermi liquid state whose characteristic energy scale T_FL collapses near Bc in a manner indicative of a magnetic field-tuned quantum critical point. A direct connection is established with the process of Kondo singlet formation, which yields a broad maximum in the magnetic susceptibility as a function of temperature in weak magnetic fields that evolves progressively into a sharper transition at Bc as T -> 0.