Nodal-line semimetals (NLSs) represent a new type of topological semimetallic beyond Weyl and Dirac semimetals in the sense that they host closed loops or open curves of band degeneracies in the Brillouin zone. Parallel to the classification of type-I and type-II Weyl semimetals, there are two types of NLSs. The conventional NLS phase, in which the two bands forming the nodal line have opposite signs for their slopes along any direction perpendicular to the nodal line, has been proposed and realized in many compounds, whereas the exotic type-II NLS is very rare. Our first-principles calculations show that Mg$_3$Bi$_2$ is a material candidate that hosts a single type-II nodal loop around $Gamma$. The band crossing is close to the Fermi level and the two crossing bands have the same sign in their slopes along the radial direction of the loop, indicating the type-II nature of the nodal line. Spin-orbit coupling generates only a small energy gap ($sim$35 meV) at the nodal points and does not negate the band dispersion of Mg$_3$Bi$_2$ that yields the type-II nodal line. Based on this prediction we have synthesized Mg$_3$Bi$_2$ single crystals and confirmed the presence of the type-II nodal lines in the material. Our angle-resolved photoemission spectroscopy (ARPES) measurements agree well with our first-principles results and thus establish Mg$_3$Bi$_2$ as an ideal materials platform for studying the exotic properties of type-II nodal line semimetals.