In recent years, the development of nanophotonic devices has presented a revolutionary means to manipulate light at nanoscale. Recently, artificial neural networks (ANNs) have displayed powerful ability in the inverse design of nanophotonic devices. However, there is limited research on the inverse design for modeling and learning the sequence characteristics of a spectrum. In this work, we propose a novel deep learning method based on an improved recurrent neural networks to extract the sequence characteristics of a spectrum and achieve inverse design and spectrum prediction. A key feature of the network is that the memory or feedback loops it comprises allow it to effectively recognize time series data. In the context of nanorods hyperbolic metamaterials, we demonstrated the high consistency between the target spectrum and the predicted spectrum, and the network learned the deep physical relationship concerning the structural parameter changes reflected on the spectrum. Moreover, the proposed model is capable of predicting an unknown spectrum based on a known spectrum with only 0.32% mean relative error. We propose this method as an effective and accurate alternative to the application of ANNs in nanophotonics, paving way for fast and accurate design of desired devices.