We present results on the masses of the low-lying baryons using ten ensembles of gauge configurations with $N_f =2+1+1$ dynamical twisted mass fermions, at three values of the lattice spacing, spanning a pion mass range from about 210 MeV to about 430 MeV. The strange and charm quark masses are tuned to approximately their physical values. We examine isospin symmetry breaking effects on the baryon mass and the dependence on the lattice spacing. After taking the continuum limit we use chiral perturbation theory to extrapolate to the physical vlaue of the pion mass for all forty baryons. We provide predictions for the masses of doubly and triply charmed baryons that have not yet been measured experimentally.