We present a search for a light sterile neutrino using three years of atmospheric neutrino data from the DeepCore detector in the energy range of approximately $10-60~$GeV. DeepCore is the low-energy sub-array of the IceCube Neutrino Observatory. The standard three-neutrino paradigm can be probed by adding an additional light ($Delta m_{41}^2 sim 1 mathrm{ eV^2}$) sterile neutrino. Sterile neutrinos do not interact through the standard weak interaction, and therefore cannot be directly detected. However, their mixing with the three active neutrino states leaves an imprint on the standard atmospheric neutrino oscillations for energies below 100 GeV. A search for such mixing via muon neutrino disappearance is presented here. The data are found to be consistent with the standard three neutrino hypothesis. Therefore we derive limits on the mixing matrix elements at the level of $|U_{mu4}|^2 < 0.11 $ and $|U_{tau4}|^2 < 0.15 $ (90% C.L.) for the sterile neutrino mass splitting $Delta m_{41}^2 = 1.0$ eV$^2$.