The historic first joint detection of both gravitational wave and electromagnetic emission from a binary neutron star merger cemented the association between short gamma-ray bursts (SGRBs) and compact object mergers, as well as providing a well sampled multi-wavelength light curve of a radioactive kilonova (KN) for the first time. Here we compare the optical and near-infrared light curves of this KN, AT2017gfo, to the counterparts of a sample of nearby (z < 0.5) SGRBs to characterize their diversity in terms of their brightness distribution. Although at similar epochs AT2017gfo appears fainter than every SGRB-associated KN claimed so far, we find three bursts (GRBs 050509B, 061201 and 080905A) where, if the reported redshifts are correct, deep upper limits rule out the presence of a KN similar to AT2017gfo by several magnitudes. Combined with the properties of previously claimed KNe in SGRBs this suggests considerable diversity in the properties of KN drawn from compact object mergers, despite the similar physical conditions that are expected in many NS-NS mergers. We find that observer angle alone is not able to explain this diversity, which is likely a product of the merger type (NS-NS versus NS-BH) and the detailed properties of the binary (mass ratio, spins etc). Ultimately disentangling these properties should be possible through observations of SGRBs and gravitational wave sources, providing direct measurements of heavy element enrichment throughout the Universe.