Most electronic properties of metals are determined solely by the low-energy states around the Fermi level, and for topological metals/semimetals, these low-energy states become distinct because of their unusual energy dispersion and emergent pseudospin degree of freedom. Here, we propose a class of materials which are termed as quadratic contact point (QCP) semimetals. In these materials, the conduction and valence bands contact at isolated points in the Brillouin zone, around which the band dispersions are quadratic along all three directions. We show that in the absence/presence of spin-orbit coupling, there may exist triply/quadruply-degenerate QCPs that are protected by the crystalline symmetry. We construct effective models to characterize the low-energy fermions near these QCPs. Under strong magnetic field, unlike the usual 3D electron gas, there appear unconventional features in the Landau spectrum. The QCP semimetal phase is adjacent to a variety of topological phases. For example, by breaking symmetries via Zeeman field or lattice strain, it can be transformed into a Weyl semimetal with Weyl and double Weyl points, a Z2 topological insulator/metal, or a Dirac semimetal. Via first-principles calculations, we identify realistic materials Cu2Se and RhAs3 as candidates for QCP semimetals.