The incorporation of Wilson lines leads to an extension of the modular symmetries of string compactification beyond $mathrm{SL}(2,mathbb Z)$. In the simplest case with one Wilson line $Z$, Kahler modulus $T$ and complex structure modulus $U$, we are led to the Siegel modular group $mathrm{Sp}(4,mathbb Z)$. It includes $mathrm{SL}(2,mathbb Z)_Ttimesmathrm{SL}(2,mathbb Z)_U$ as well as $mathbb Z_2$ mirror symmetry, which interchanges $T$ and $U$. Possible applications to flavor physics of the Standard Model require the study of orbifolds of $mathrm{Sp}(4,mathbb Z)$ to obtain chiral fermions. We identify the 13 possible orbifolds and determine their modular flavor symmetries as subgroups of $mathrm{Sp}(4,mathbb Z)$. Some cases correspond to symmetric orbifolds that extend previously discussed cases of $mathrm{SL}(2,mathbb Z)$. Others are based on asymmetric orbifold twists (including mirror symmetry) that do no longer allow for a simple intuitive geometrical interpretation and require further study. Sometimes they can be mapped back to symmetric orbifolds with quantized Wilson lines. The symmetries of $mathrm{Sp}(4,mathbb Z)$ reveal exciting new aspects of modular symmetries with promising applications to flavor model building.