Electron evaporation plays an important role in the electron temperature evolution and thus expansion rate in low-density ultracold plasmas. In addition, evaporation is useful as a potential tool for obtaining colder electron temperatures and characterizing plasma parameters. Evaporation theory has been developed for atomic gases and has been applied to a one-component plasma system. We numerically investigate whether such an adapted theory is applicable to ultracold neutral plasmas. We find that it is not due to the violation of fundamental assumptions of the model. The details of our calculations are presented as well as a discussion of the implications for a simple description of the electron evaporation rate in ultracold plasmas.