Femtolensing of gamma ray bursts (GRBs) has been put forward as an exciting possibility to probe exotic astrophysical objects with masses below $10^{-13}$ solar masses such as small primordial black holes or ultra-compact dark matter minihalos, made up for instance of QCD axions. In this paper we critically review this idea, properly taking into account the extended nature of the source as well as wave optics effects. We demonstrate that most GRBs are inappropriate for femtolensing searches due to their large sizes. This removes the previous femtolensing bounds on primordial black holes, implying that vast regions of parameter space for primordial black hole dark matter are not robustly constrained. Still, we entertain the possibility that a small fraction of GRBs, characterized by fast variability can have smaller sizes and be useful. However, a large number of such bursts would need to be observed to achieve meaningful constraints. We study the sensitivity of future observations as a function of the number of detected GRBs and of the size of the emission region.