Six high-entropy rare earth tetraborides of the tetragonal UB4-prototyped structure have been successfully synthesized for the first time. The specimens are prepared from elemental precursors via high-energy ball mill and in-situ reactive spark plasma sintering. The sintered specimens are >98% in relative densities without detectable oxide impurities (albeit the presence of minor hexaborides in some compositions). No detectable secondary phase is observed in the composition (Y$_{0.2}$Nd$_{0.2}$Sm$_{0.2}$Gd$_{0.2}$Tb$_{0.2}$)B$_{4}$, which is proven homogeneous at both microscale and nanoscale. The Vickers microhardness are determined to be ~13-15 GPa at a standard indentation load of 9.8 N. A scientifically interesting observation is represented by the anisotropic lattice distortion from the rule-of-mixture averages. This work expands the family of high-entropy ceramics via fabricating a new class of high-entropy borides with a unique tetragonal quasi-layered crystal structure.