Spontaneous decays of small, hot silver cluster anions Ag$_{n}$, $n=4-7$ have been studied using one of the rings of the Double ElectroStatic Ion Ring ExpEriment (DESIREE). Observation of these decays over very long time scales is possible due to the very low residual gas pressure ($sim10^{-14}$) and cryogenic (13 K) operation of DESIREE. The yield of neutral particles from stored beams of Ag$_{6}$ and Ag$_{7}$ anions were measured for 100 milliseconds and were found to follow single power law behaviour with millisecond time scale exponential cut-offs. The Ag$_{4}$ and Ag$_{5}$ anions were stored for 60 seconds and the observed decays show two-component power law behaviors. We present calculations of the rate constants for electron detachment from, and fragmentation of Ag$_{4}$ and Ag$_{5}$. In these calculations, we assume that the internal energy distribution of the clusters are flat and with this we reproduce the early steep parts of the experimentally measured decay curves for Ag$_{4}$ and Ag$_{5}$, which extends to tens and hundreds of milliseconds, respectively. The fact that the calculations reproduce the early slopes of Ag$_{4}$ and Ag$_{5}$, which differ for the two cases, suggests that it is the changes in fragmentation rates with internal cluster energies of Ag$_{4}$ and Ag$_{5}$ rather than conditions in the ion source that determines this behavior. Comparisons with the measurements strongly suggest that the neutral particles detected in these time domains originate from Ag$_{4} rightarrow$ Ag$_{3}+$ Ag and Ag$_{5}rightarrow$ Ag$_{3}+$ Ag$_{2}$ fragmentation processes.