The PADME experiment, at the Laboratori Nazionali di Frascati (LNF), in Italy, will search for invisible decays of the hypothetical dark photon via the process $e^+e^-rightarrow gamma A$, where the $A$ escapes detection. The dark photon mass range sensitivity in a first phase will be 1 to 24 MeV. We report here on measurement and simulation studies of the performance of the Small-Angle Calorimeter, a component of PADMEs detector dedicated to rejecting 2- and 3-gamma backgrounds. The crucial requirement is a timing resolution of less than 200 ps, which is satisfied by the choice of PbF$_2$ crystals and the newly released Hamamatsu R13478UV photomultiplier tubes (PMTs). We find a timing resolution of 81 ps (with double-peak separation resolution of 1.8 ns) and a single-crystal energy resolution of 5.7%/$sqrt{E}$ with light yield of 2.07 photo-electrons per MeV, using 100 to 400 MeV electrons at the Beam Test Facility of LNF. We also propose the investigation of a two-PMT solution coupled to a single PbF$_2$ crystal for higher-energy applications, which has potentially attractive features.