Ultraviolet (UV) plasmonics aims at combining the strong absorption bands of molecules in the UV range with the intense electromagnetic fields of plasmonic nanostructures to promote surface-enhanced spectroscopy and catalysis. Currently, aluminum is the most widely used metal for UV plasmonics, and is generally assumed to be remarkably stable thanks to its natural alumina layer passivating the metal surface. However, we find here that under 266 nm UV illumination, aluminum can undergo a dramatic photocorrosion in water within a few tens of seconds and even at low average UV powers. This aluminum instability in water environments critically limits the UV plasmonics applications. We show that the aluminum photocorrosion is related to the nonlinear absorption by water in the UV range leading to the production of hydroxyl radicals. Different corrosion protection approaches are tested using scavengers for reactive oxygen species and polymer layers deposited on top of the aluminum structures. Using optimized protection, we achieve a ten-fold increase in the available UV power range leading to no visible photocorrosion effects. This technique is crucial to achieve stable use of aluminum nanostructures for UV plasmonics in aqueous solutions.