We report the magneto-transport properties of CaAl$_4$ single crystals with $C2/m$ structure at low temperature. CaAl$_4$ exhibits large unsaturated magnetoresistance $sim$3000$%$ at 2.5 K and 14 T. The nonlinear Hall resistivity is observed, which indicates the multi-band feature. The first-principles calculations show the electron-hole compensation and the complex Fermi surface in CaAl$_4$, to which the two-band model with over-simplified carrier mobility cant completely apply. Evident quantum oscillations have been observed with B//c and B//ab configurations, from which the nontrivial Berry phase is extracted by the multi-band Lifshitz-Kosevich formula fitting. An electron-type quasi-2D Fermi surface is found by the angle-dependent Shubnikov-de Haas oscillations, de Haas-van Alphen oscillations and the first-principles calculations. The calculations also elucidate that CaAl$_4$ owns a Dirac nodal line type band structure around the $Gamma$ point in the $Z$-$Gamma$-$L$ plane, which is protected by the mirror symmetry as well as the space inversion and time reversal symmetries. Once the spin-orbit coupling is included, the crossed nodal line opens a negligible gap (less than 3 meV). The open-orbit topology is also found in the electron-type Fermi surfaces, which is believed to help enhance the magnetoresistance observed.