Soft Gamma-ray Repeaters (SGRs) and Anomalous X-ray Pulsars (AXPs) are interpreted as young highly magnetized neutron stars (NSs). Their X-ray luminosity in quiescence, exceeding 10^{35} erg s^{-1} cannot be explained as due to cooling of a highly magnetized NS, but requires as an extra heat source the decay of its magnetic field (MF). We study numerically the coupled evolution of the MF, temperature and spin period under the assumption that the currents maintaining the field are confined in the crust of the star. The decay of the field depends on the field strength itself (Hall-drift), on the temperature and injects heat into the star, but is controlled by neutrino emission. Finally we consider the spin down from magnetic dipole braking with this decaying field to track the long term evolution. We find reasonable initial conditions for the MF strength and structure to explain their current observational values both of their rotational period, its time derivative and the X-ray luminosity of AXPs and SGRs.the X-ray luminosity of AXPs and SGRs.